AI crawlers, robots.txt and llms.txt: what the vendors actually document

AI crawlers are automated clients that AI vendors use to fetch web pages, each identified by a user-agent token such as GPTBot or PerplexityBot. robots.txt is the only file those vendors document as controlling them. llms.txt is a separate community proposal, and we found no vendor document saying an engine reads it.

This is the technical lesson in the series, and it is the one where the gap between what gets repeated and what anyone has published is widest. So the rule for this page is narrow: a claim about a crawler appears here only if we opened the operator's own documentation and read it. Everything below was read on 2026-09-20, and the URL is in the table. Where we have no source, the section says so instead of filling the space.

None of this is about getting cited. Being fetchable is a precondition, not a cause, and the two get conflated constantly. What we measure, and what a measurement can and cannot support, is on AI visibility tracking and in our methodology.

The crawlers their operators document

This table covers the four vendors whose bots our own robots.txt names. It is not a complete census of AI crawlers on the web. Rows for other operators are missing because we did not read their documentation in this pass, not because their bots do not exist.

User agent	Operator	What the operator says it is for	Source, read 2026-09-20
GPTBot	OpenAI	"used to make our generative AI foundation models more useful and safe"; crawls content that may go into training	developers.openai.com/api/docs/bots
OAI-SearchBot	OpenAI	"used to surface websites in search results in ChatGPT's search features"	developers.openai.com/api/docs/bots
ChatGPT-User	OpenAI	visits a page when a user asks ChatGPT or a custom GPT a question	developers.openai.com/api/docs/bots
OAI-AdsBot	OpenAI	"used to validate the safety of web pages submitted as ads" on ChatGPT	developers.openai.com/api/docs/bots
PerplexityBot	Perplexity	surfaces and links sites in Perplexity search results; "not used to crawl content for AI foundation models"	docs.perplexity.ai/guides/bots
Perplexity-User	Perplexity	fetches a page to answer a question a user just asked	docs.perplexity.ai/guides/bots
ClaudeBot	Anthropic	collects web content "that could potentially contribute to their training"	support.claude.com, article 8896518
Claude-User	Anthropic	accesses sites when an individual asks Claude a question	support.claude.com, article 8896518
Claude-SearchBot	Anthropic	"navigates the web to improve search result quality for users"	support.claude.com, article 8896518
Google-Extended	Google	a control token for whether crawled content trains Gemini models and grounds answers in Gemini apps	developers.google.com/search/docs/crawling-indexing/google-common-crawlers
Googlebot	Google	the Search crawler; Search is what feeds Google's AI features	developers.google.com/search/docs/crawling-indexing/google-common-crawlers

Two things in that table are easy to miss.

Google-Extended is not a crawler. Google's page states it "doesn't have a separate HTTP request user agent string", and that the fetching is done by existing Google agents while the token works as a control. You will never see it in a log. It only exists as a line in robots.txt.

OAI-AdsBot only visits pages that someone submitted as an ad. It is in the table because OpenAI documents it, not because it has anything to do with organic visibility.

The bots do three different jobs, and the difference decides the trade-off

Sort the table by what the fetch is for and it becomes much easier to reason about.

Job	Bots	What blocking it costs you
Model training	GPTBot, ClaudeBot, Google-Extended	Your content is excluded from future training data. No documented effect on whether an engine retrieves you today
Search and retrieval index	OAI-SearchBot, PerplexityBot, Claude-SearchBot	The vendor says your site becomes less likely, or unable, to appear in that product's answers
A user asked right now	ChatGPT-User, Perplexity-User, Claude-User	The page cannot be fetched when a person pastes your URL or asks about you, though robots.txt may not be obeyed here anyway

The middle row is where the real decision sits. OpenAI states that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers", while still appearing as navigational links. Perplexity recommends allowing PerplexityBot for the same reason. Anthropic says disabling Claude-SearchBot prevents indexing for search. These are the operators describing their own products, which is the strongest kind of evidence available on this question and still not an outcome you can bank on.

The bottom row comes with a caveat the vendors write themselves. OpenAI says robots.txt rules "may not apply" to ChatGPT-User because the action is user-initiated. Perplexity is blunter: "this fetcher generally ignores robots.txt rules". If your reason for blocking is that you do not want your page read at all, robots.txt is the wrong instrument, and you are looking at authentication or a firewall rule instead.

robots.txt: only one group applies

This is the part that breaks quietly, because a wrong robots.txt returns 200 and looks fine.

Under RFC 9309, the Robots Exclusion Protocol standard published in September 2022, a crawler finds the group matching its own product token and obeys that group. Groups do not merge. Google's own documentation says it plainly: "Only one group is valid for a particular crawler", and user-agent-specific groups and the * group "are not combined".

So a file that looks like this does not do what most people reading it assume:

User-Agent: *
Allow: /
Disallow: /private/

User-Agent: GPTBot
Allow: /

GPTBot matches its own group. It obeys Allow: / and never sees Disallow: /private/. Naming a bot in order to be explicit and welcoming has quietly given it more access than the wildcard group gives everyone else. We know this failure mode well, because our own file had it until we wrote this lesson. More on that below.

Allowing and blocking are both defensible positions, and the honest framing is a trade rather than a recommendation:

	Allow the retrieval bots	Block them
Visibility in that engine	Eligible, by the vendor's own description. Eligible is not the same as cited	The vendor states you will not appear, or will appear less
Control over your text	Your prose can be summarised and quoted in an answer you never see	Reduced for that engine, and unchanged for every engine you did not list
Bandwidth and cost	Additional automated traffic, some of it with no referral back	Lower
Training exposure	Governed by a different token, not by the retrieval bot	Also governed by a different token
Reversibility	Immediate to change; OpenAI and Perplexity both say updates take roughly 24 hours to take effect	Same

Publishers with licensing arrangements, paywalls or archives to protect land on one side of this. Businesses trying to be found by buyers usually land on the other. Neither answer is a technical fact.

Structured data: whose surface are you talking about?

Google publishes a clear position for its own surface. Its guidance for site owners states that there are "no additional requirements to appear in AI Overviews or AI Mode", and that "there's also no special schema.org structured data that you need to add" (Google Search Central, AI features and your website, read 2026-09-20). Google AI Overviews and AI Mode are outside what we measure, so we have nothing of our own to add to that, in either direction.

What that statement does not do is cover ChatGPT, Gemini's own apps or Perplexity. We have no measurement of our own on whether schema markup changes how those engines retrieve or quote a page, and none of the three publishes guidance on it. A Google statement about Google Search is not evidence about an engine Google does not operate. It is also not evidence that markup does nothing elsewhere. Both of those readings appear in published articles on this topic and neither is supported.

Where structured data still earns its place is the ordinary one: it makes a page eligible for rich results in Search, and it describes entities in a form a parser can read without guessing. That case has not changed. What has changed is the volume of writing asserting a causal link between markup and AI citations, which we have not seen anyone measure.

llms.txt

Three separate questions get answered as one here. Keeping them apart is most of the work.

What it is

A proposal by Jeremy Howard, published at llmstxt.org on 2024-09-03 and revised to v2 on 2026-08-10. It defines a markdown file at /llms.txt giving background, guidance and links to detailed markdown versions of a site's pages, on the argument that agents are better served by clean text than by HTML full of navigation and scripts. It is a convention, not a standard in the sense that robots.txt is one. Nothing enforces it and no registry governs it.

Who claims support

The proposal itself is careful about this. It notes that "the AI labs themselves publish llms.txt files for their own developer docs", naming OpenAI, Anthropic and Gemini. That is a claim about labs publishing the file, not about their crawlers reading yours, and the distinction gets lost constantly. Perplexity's own developer documentation, for example, shows a banner pointing agents at its /llms.txt index. The same site's crawler documentation says nothing about PerplexityBot reading anybody else's.

Documentation platforms generate the file automatically, which explains a large share of the files in existence. Generation by a vendor's docs tooling is not consumption by that vendor's crawler.

What evidence there is

Google is the only operator we found with a published position, and it is negative for its own surface. Its guide to generative AI features in Search lists llms.txt under things "you can ignore for Google Search", says the file is not needed to appear there, and states that maintaining one will "neither harm nor help your site's visibility or rankings" because "Google Search ignores them" (Google Search Central, Google's guide to optimizing for generative AI features on Google Search, last updated 2026-07-10, read 2026-09-20).

That is Google Search. It is not a statement about ChatGPT, Claude or Perplexity, and it cannot be quoted as one.

For those three, here is how we looked and what we found. We read OpenAI's bots documentation, Perplexity's bots guide and Anthropic's crawler support article on 2026-09-20 and searched each page for the string. Anthropic's article does not contain it. The matches on the OpenAI and Perplexity pages are links to those companies' own documentation indexes, not statements about their crawlers reading other sites. As of 2026-09-20 we could not find any AI engine operator publicly stating that it reads llms.txt on third-party sites. Absence of a published statement is not proof that no engine reads it. It is the reason we will not tell you it works.

One outside measurement is worth reporting, with its provenance attached, because it is the only attempt at the question we found. Ahrefs examined roughly 38,000 domains carrying a valid llms.txt and reported that "97% received zero requests for it in May 2026" (Ryan Law, What is llms.txt, and should you care about it?, ahrefs.com, updated 2026-06-15, read 2026-09-20). That is their number and their method, not ours, and we have not reproduced it.

There is also a genuine inconsistency inside Google worth naming rather than smoothing over. While Search documentation says the file is ignored, the Chrome team ships an llms.txt check in Lighthouse's agentic-browsing audits. That audit only flags a server error and marks a missing file as not applicable, since "providing the file is optional at the moment" (Chrome for Developers, Lighthouse llms.txt audit, read 2026-09-20). A lint rule from one team is not an endorsement from another.

Our own setup, including the bug we found in it

We publish both files. Reading the source is more useful than reading a recommendation, so here is what ours actually do, and what one of them got wrong until this lesson was written.

robots.txt is generated by src/app/robots.ts. Until we wrote this page it emitted a wildcard group allowing everything except /api/, followed by a second group that named GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Google-Extended, ClaudeBot and Claude-SearchBot with a bare Allow: /. The intent was to be explicit about welcoming AI crawlers.

It had the exact defect described three sections up. Because groups do not merge, each of those seven bots matched its own group, obeyed the single Allow: / line in it, and never saw Disallow: /api/. Every unnamed client did see it. Nothing leaked: /api/ holds webhook, auth and billing endpoints that are linked from nowhere and mostly reject GET. The problem is simpler and worse to leave in place. The file did not express what it was written to express, and it said the opposite of what its author believed it said.

Two further oddities in that group were symptoms of the same confusion. Claude-User and Perplexity-User were not on the list, so Anthropic's and Perplexity's user-initiated fetchers were governed by a different group from their sibling bots, with no reason behind the split. And Google-Extended sat in an allow group where it changed nothing, since allowed is already the default for it. A group of rules that grants no access, splits siblings at random and inverts one directive is not a configuration. It is a comment that the parser takes seriously.

So we deleted it. The file now emits one group and nothing else:

User-agent: *
Allow: /
Disallow: /api/

Deleting beat the obvious patch, which would have been to repeat Disallow: /api/ inside the named group. That patch restores the intended behaviour and leaves the trap armed: the next person to add a path to the wildcard group has to remember to add it twice, and will not, because nothing in the file says so. The named group was never doing any work in the first place. Every bot in it was already allowed by the wildcard, so removing it changes the access those seven bots have by nothing at all, while removing the only mechanism by which the file could drift out of agreement with itself. Rules that exist to signal a position, rather than to change one, are worth deleting on principle in a file a machine parses literally.

Pages that should stay out of the index still use a meta robots tag rather than a Disallow, on the reasoning that a blocked crawler never reads the noindex.

We found this while writing this lesson, not before publishing it, and we are leaving the account in rather than quietly shipping the fix. A page about technical setup written by someone whose own file was wrong is worth more than one that implies the author has never made the mistake. This particular mistake is close to invisible: the file returns 200, every validator accepts it, and each line in it is individually valid.

Dropping Google-Extended from the file leaves our position on it unchanged, since the token defaults to allowed and we were allowing it. What it does change is that the position is now implicit, so it is worth stating here instead. Google documents that token as governing both Gemini training and grounding in Gemini apps and on Vertex AI. Our measurement pipeline queries Gemini with Google Search grounding, and whether the token reaches the API path our pipeline uses is something we have not tested. We are not going to assume it either way.

llms.txt is served by src/app/llms.txt/route.ts as a static route. It carries a one-paragraph description of the company, the product pages with their terms, the policy pages and a contact address. It does not list the articles in this series, because it was written when the site was three product pages.

We keep it for one reason: it costs a few minutes a quarter, it is a readable summary of what we sell, and Google states in its own documentation that it does no harm. We do not claim it produces anything. If we later get evidence that some engine fetches it, this paragraph is where that evidence will go.

What we cannot tell you

We do not have AI crawler hit data for this site. Reading the edge logs on our host is not set up, and we are not going to estimate which bots visit us or how often from anything else. A number like that would be the easiest thing in this article to invent and the hardest for you to check, which is precisely why it is absent. When we have real log access, the counts will appear here with the date range attached.

We also have no controlled test of our own linking any technical setting to a change in citation behaviour. The measurement pipeline described in our methodology measures answers, not crawlers. A page that claims otherwise is selling you something.

Questions

Which AI crawler user agents should I know about? For the engines covered here: GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot from OpenAI; PerplexityBot and Perplexity-User from Perplexity; ClaudeBot, Claude-User and Claude-SearchBot from Anthropic; and Google-Extended as a control token rather than a real crawler. Each vendor publishes the list and the full user-agent strings, and those pages are linked in the table above.

Does blocking GPTBot stop ChatGPT from citing my site? Not according to OpenAI's documentation. GPTBot is described as the training crawler, and search visibility in ChatGPT is governed by OAI-SearchBot. They are independent settings, and OpenAI says so explicitly.

Does Google use llms.txt? No. Google's own guidance states that Google Search ignores the file and that keeping one will neither harm nor help visibility or rankings there. That statement covers Google Search only.

Do ChatGPT or Perplexity read llms.txt? Neither company documents doing so. We read both vendors' crawler pages on 2026-09-20 and found no such statement, which is not the same as proof that nothing reads the file. We treat the question as open and unsupported rather than settled.

Should I create an llms.txt file? It is cheap and, by Google's account, harmless on that surface. Create it if you want a clean summary of your site in one place. Do not create it expecting a measurable result, because nobody has published one.

Is schema markup required for AI search? Google says no additional structured data is needed for its own AI features. For ChatGPT, Gemini's apps and Perplexity there is no published guidance and no measurement we trust, including ours. Use structured data for the reasons that were always valid, and treat any claim beyond that as untested.

How do I check whether an AI crawler is really who it says it is? Each of these vendors publishes IP ranges for verification: OpenAI and Perplexity as JSON files linked from their bots pages, Anthropic at a published bots endpoint, Google through its standard crawler verification. A user-agent string on its own is trivially forged.

Will getting the technical setup right get me cited? No. It decides whether a crawler can reach the page. Whether an engine then names you in an answer is a separate question, it varies between runs of the same question, and the only way to know is to measure it. That is what AI visibility tracking does.

What to do next

Open your own robots.txt and check two things. First, whether any group names a specific bot, because if one does, that bot is ignoring every rule in your wildcard group. Second, whether the retrieval bots and the training bots are treated as the same decision, because the vendors do not treat them that way and you probably do not want to either.

Then leave it alone. The technical layer is a gate, and a gate that is already open cannot be opened further. The question of whether an engine names you when a buyer asks is answered by measurement, not configuration, and the terms in that measurement are defined in what GEO and AEO actually mean.