RedClawLabs

Ad math calculators

AI crawler check

The AI crawler check reads three public files on your site (the home page, robots.txt and llms.txt) and reports whether 10 crawlers from ChatGPT, Claude, Perplexity and Google may fetch your home page, and whether that page carries text and structured data they can read. Free, no sign-in, about 5 seconds.

Check · reads three public files: home page, robots.txt, llms.txt

Enter your domain

e.g. example.com, without https://

Reading

Checks passed

— / 9

robots.txt
—
llms.txt
—
Home page response
—

Type a domain and run it. About 5 seconds, no sign-in.

Check by check

CheckResultWhat we read
robots.txt is readable, or clearly absent——
AI search crawlers may fetch the home page——
User-triggered fetchers may read the home page——
No noindex or nosnippet on the home page——
Body text is in the raw HTML (at least 500 characters)——
JSON-LD structured data on the home page——
Title and meta description present——
An h1 is present——
A canonical is set——

Crawler by crawler

CrawlerCompanyUsed forMay fetch the home page
OAI-SearchBotOpenAISearch index—
ChatGPT-UserOpenAIUser-triggered—
GPTBotOpenAIModel training—
Claude-SearchBotAnthropicSearch index—
Claude-UserAnthropicUser-triggered—
ClaudeBotAnthropicModel training—
PerplexityBotPerplexitySearch index—
Perplexity-UserPerplexityUser-triggered—
GooglebotGoogleSearch index—
Google-ExtendedGoogleModel training—

llms.txt draft

After a check, an editable draft appears here, ready to download.

The domain you type is sent to our server so it can read those three public files; it is not written to a database or sent to analytics. To prevent abuse we keep a hashed IP and a count. The site-wide analytics are described in the privacy policy.

Three kinds of AI crawler, three different consequences

Each vendor splits its crawlers by purpose, and robots.txt can treat them separately. The usual mistake is blocking the search crawler along with the training one.

Used for Crawlers If you block it
Search index OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot Your content does not appear in that engine's search answers
User-triggered ChatGPT-User, Claude-User, Perplexity-User The assistant cannot read your page when a user pastes the URL or asks it to look
Model training GPTBot, ClaudeBot, Google-Extended Content is kept out of future model training; search is not affected

Three points that are easy to get wrong, each stated on the vendor's own page (sources below):

  • Blocking GPTBot does not remove you from ChatGPT search. OpenAI says each setting is independent of the others; ChatGPT search is governed by OAI-SearchBot.
  • Google-Extended does not affect inclusion or ranking in Google Search. It only controls use of your content for Gemini training and grounding.
  • User-triggered fetchers may not follow robots.txt. OpenAI says the rules may not apply because a user started the request; Perplexity says Perplexity-User generally ignores robots.txt.

How each check is decided

Check How it is decided
robots.txt Per RFC 9309: a crawler with its own group obeys only that group, otherwise *; inside a group the longest matching rule wins, and Allow wins a tie. A 404 means everything is allowed; a 5xx is reported as unknown
May fetch the home page The rules above, applied to the path / for each crawler
noindex, nosnippet The home page's meta robots and its X-Robots-Tag header
Text in the raw HTML Characters left in the HTML the server returns after scripts, styles and tags are removed. This tool does not run JavaScript
Structured data The @type values in the home page's JSON-LD
Title, description, h1, canonical Present in the raw HTML or not
llms.txt Whether /llms.txt at the root returns plain text; not counted as a check

Whether to allow the training crawlers (GPTBot, ClaudeBot, Google-Extended) is your decision. The table reports them as they are and does not count them.

What this tool cannot see

  • Firewall and CDN blocks. robots.txt may allow a crawler that your host or CDN turns away earlier. That does not show here; check your server logs for whether those crawlers get a 200 or a 403.
  • Pages other than the home page. Only the home page and the two root files are read.
  • Whether an AI actually names you. Being crawlable is the precondition, not the result. The result only shows when you ask the questions your buyers ask and count the answers: the free brand check asks ChatGPT 3 questions once, and GEO monitoring asks three engines every week.

Should you publish an llms.txt

llms.txt is a proposal from llmstxt.org: a Markdown file at the site root with an H1 for the site name, a blockquote summary, and H2 lists of links to the pages that matter. It is not a standard the way robots.txt is, and having one or not does not change whether any crawler above may fetch your site, so this tool does not count it as a check.

Publishing one costs little: the draft above is assembled from your home page's title, description and internal links. Edit it before you download it. Drop the links that do not matter and add a line of notes to the ones you keep.

Questions

If I block GPTBot, can ChatGPT search still find me?

Yes. GPTBot is the training crawler; ChatGPT search uses OAI-SearchBot, and OpenAI says each setting is independent of the others. To appear in ChatGPT search answers, the one to allow is OAI-SearchBot.

robots.txt says allowed. Does that mean AI crawlers get through?

Not necessarily. A firewall at your host or CDN can turn a crawler away before robots.txt matters, and this tool cannot see that layer. Check your server logs for whether those crawlers receive a 200 or a 403.

Will I miss out on AI citations without an llms.txt?

You are not excluded for lacking one. llms.txt is a proposal and does not change whether a crawler may fetch your site. It is cheap to publish, but do not expect it to earn citations by itself.

Why check for text in the raw HTML?

This tool does not run JavaScript; it measures the HTML the server returns. A crawler that does not run JavaScript sees exactly that. If your text only appears after scripts run, such a crawler receives an empty shell.

Do you record the domain I enter?

No. The domain is sent to our server to read three public files (the home page, robots.txt and llms.txt) and is discarded after the check. It is not written to a database or sent to analytics. To prevent abuse we keep a hashed IP and a count of checks.

Sources

Crawler names, purposes and the robots.txt rules were checked against these official pages, opened on 2026-11-12.


Crawlable is the precondition. Whether an AI names you only shows when you ask.

The free check runs 3 questions on ChatGPT only, needs a sign-in, and is limited to one per account per month.