llms.txt, Web Bot Auth, and Crawler Policy: The Access Layer of AI Search

The access layer is the part of AI search work that gets decided by whoever last touched the firewall. For most eCommerce teams it consists of three things that behave very differently: a file everyone talks about and few systems read, a set of infrastructure rules that quietly decide whether AI crawlers reach your catalog at all, and a signed-request mechanism that determines which automated visitors get treated as legitimate. Ranked by attention received, llms.txt leads. Ranked by consequence for a commerce catalog, it comes last.

That inversion is worth an afternoon of your time, because the layer nobody is discussing is the one that can remove a store from generated answers entirely.

The expectation: llms.txt as the AI-era robots.txt

The reasoning is sound on its face. Search engines read robots.txt, so AI systems should read an equivalent file. A curated Markdown index of your most important pages sounds like exactly what a model would want: less noise, fewer tokens wasted on navigation and cookie banners, a clean statement of what the site is and where the substance lives.

The proposal itself is coherent. It came from Jeremy Howard of Answer.AI in September 2024, and it was framed as a way to help language models use a website at inference time. Serious companies ship one. Publishing a file takes under an hour. Every part of that adds up to a reasonable conclusion, which is why so many eCommerce teams have it on a roadmap.

What the evidence actually shows

Google’s position is documented rather than inferred. Its guidance on AI features and your website states that no new machine-readable files, AI text files, or special schema.org structured data are needed to appear in those features, and its guide to optimizing for generative AI search was updated in June 2026 with the same message about special files. Google Search team members have said the same on the record since 2025.

The independent data points the same way. A study of 300,000 domains by SE Ranking, widely reported this year, found adoption at roughly ten percent and ran a model to test whether the presence of an llms.txt file correlated with citation frequency; removing the file from the model improved its predictive accuracy, meaning the variable behaved as noise rather than signal. Separate research reported by Originality.ai found that adoption grew sharply while the large majority of published files were never fetched by AI systems at all.

There is one genuine complication, and it sits inside Google. Chrome’s Lighthouse added an agentic browsing audit in May 2026 that checks for llms.txt, at the same time Search Central was telling site owners the file is not needed. Both teams are consistent on their own terms, because they are describing different consumers: a search index on one side, agentic browsers and coding assistants on the other.

Which is the accurate summary. The systems that demonstrably read llms.txt today are developer-facing: coding assistants retrieving documentation, agent frameworks, and some tracking crawlers. Claims that particular consumer AI search products retrieve it are disputed between sources and unconfirmed by the vendors themselves. For a store selling products, that use case is not your use case.

Why the gap exists

Three mechanisms explain why an idea this reasonable produces so little.

A self-declared file has no verification behind it. A document in which a site states what it is about, with nothing checking the claim, has an obvious precedent in the keywords meta tag. Any system relying on it inherits a gameable input, and retrieval systems already have a stronger one: the pages themselves, crawled and evaluated.

The problem it solves is a documentation problem. Token efficiency matters enormously when an agent is reading API references to write code. A shopper query resolves against product data, reviews, and category content that the retrieval pipeline already handles. The pain llms.txt removes is real, and it is felt by developer-tools companies rather than by catalogs.

Nothing enforces it. There is no standards body behind the format and no obligation on any vendor to honour it, which is why adoption reporting is fragmented and why each provider’s behaviour has to be observed rather than assumed.

None of this makes publishing one wrong. It makes it a low-cost, low-expectation exercise that belongs nowhere near the top of a commerce roadmap.

What the access layer actually controls

Here is the layer that decides outcomes, with the specific thing each mechanism governs.

The consequential row is the second one. Policy and behaviour diverge constantly: a robots.txt that permits an AI crawler means nothing if the security layer in front of it returns a denial to the same request. Teams discover this while trying to audit their own site, when a large crawl of a Shopify store stalls on rate-limit responses that read as a content problem in a report and are actually a security-layer problem. Since Shopify added support for signed crawler requests through Web Bot Auth, that signature is what separates a complete audit from a partial one, and our walkthrough of Screaming Frog on Shopify covers that configuration in practice.

The pattern worth internalising: your stated policy lives in one system, your actual behaviour lives in another, and nobody owns the job of keeping them in agreement.

The decision nobody writes down

Underneath the technical work sits a policy question that is usually resolved by default rather than by decision: which automated visitors are welcome, and why.

The question is harder than it looks because two different activities travel under one label. Crawling for training corpora and crawling to retrieve an answer for a user asking about your category right now are separate things with separate consequences, and they are not always distinguishable by user agent. Restricting the first can restrict the second. For a commerce catalog, that trade is asymmetric in an uncomfortable direction: the content being protected is product information you publish specifically so people can find it.

Three positions are defensible, and a store should hold one of them deliberately.

Permit broadly, on the reasoning that product data exists to be discovered and that presence in generated answers is worth more than control over how a model learned about your catalog. Permit selectively, allowing retrieval-oriented crawlers while restricting those associated primarily with training corpora, accepting that the distinction is imperfect and needs review as crawler documentation changes. Or restrict broadly, which is a coherent position for brands whose editorial content is the product, and an expensive one for a store whose catalog is.

What matters more than the choice is that it exists as a written decision with a named owner. A firewall rule added during an incident is not a policy, and it will outlive everyone’s memory of why it was added.

What to check this week

Five checks, none of which need a budget.

1. Fetch your own pages as an AI crawler. Request a product page and a category page while presenting the user agents of the major AI crawlers. Record the status code and whether the response contains your full HTML. A denial or an empty body is your answer.
2. Compare stated policy against observed behaviour. Read your robots.txt, then read your CDN and WAF rules. Note every place they disagree, since the infrastructure wins.
3. Look for rate limiting on scale. Run a crawl large enough to matter and watch for responses that appear partway through rather than at the start. Partial crawls produce partial audits and confident wrong conclusions.
4. Check whether your platform supports signed crawler requests, and whether the tools you use for audits are configured to sign. This is the difference between auditing your store and auditing the part of your store your firewall felt like serving that day.
5. Write the policy down. One page: which crawlers are permitted, which are restricted, the reasoning, and who approves changes. Date it.

Run these in order. The first check ends the exercise for a surprising share of stores, because the answer arrives immediately and the work becomes a single infrastructure ticket.

Revised expectations

Ship llms.txt if you want to, with accurate expectations attached: it is inexpensive, it may serve agentic browsers and developer tooling, and no major AI search product has committed to reading it. Publish it, keep it accurate, and give it no place in a visibility plan. Documentation-heavy sites and API products have a stronger case than any catalog does.

Spend the attention instead on whether AI crawlers receive your pages, whether your infrastructure agrees with your stated policy, and whether the block-or-allow question has an owner. Those three determine whether anything else in an AI search programme can produce a result, and they are boring enough that they tend to stay unexamined for years. In AI consultancy work the access layer is where the first genuine finding usually appears, well before any content question is reached.

Frequently Asked Questions

Does llms.txt help eCommerce sites appear in AI search? 

There is no documented evidence that it does. Google states that no special machine-readable files are needed for its AI features, and independent analysis of adoption at scale found no citation effect, with most published files never fetched. The systems demonstrably reading llms.txt today are coding assistants and agent tooling rather than consumer AI search products.

Should I block AI crawlers to protect my content? 

That is a policy decision rather than a technical one, and for a commerce catalog the trade is asymmetric. Restricting crawlers can also restrict retrieval for shoppers asking about your category, because training and retrieval activity are not cleanly separable by user agent. Whatever you choose, record it as a written decision with an owner rather than leaving it to a firewall rule.

Why does my robots.txt say one thing and my logs show another? 

Because robots.txt states policy while your CDN or web application firewall decides whether a request is served. Security layers frequently deny or rate-limit automated traffic independently of what robots.txt permits. Read both, and treat the infrastructure behaviour as the real policy until it is reconciled.

What is Web Bot Auth and do I need it? 

It is a mechanism for automated requests to prove which agent they come from, rather than being judged by a user-agent string that anything can claim. On platforms that support it, signed requests are treated differently from unsigned ones, which matters both for crawlers reaching your store and for your own audit tooling completing a full crawl.

Key Takeaways

• llms.txt receives the most attention and carries the least consequence for a commerce catalog. Google states no special files are needed for its AI features, and adoption analysis at scale found the file behaving as noise rather than signal for citation.
• The consequential layer is infrastructure. A robots.txt that permits AI crawlers means nothing when a CDN or firewall denies the same request, and that divergence is the normal state rather than the exception.
• Blocking decisions are policy, not engineering. Training and retrieval crawling are not cleanly separable, so restricting one can remove a store from answers shoppers are asking for right now.
• Five checks settle the question this week, starting with fetching your own pages as an AI crawler. For many stores that first check ends the exercise and turns it into one infrastructure ticket.

The access layer rarely appears on a content calendar because it produces no deliverable anyone can show. It is also the only layer where a single unexamined rule can make every other investment in AI visibility unreadable.