The AI crawler table

You cannot write robots-ai.txt with intent until you know what each crawler is for. Most of them are not interchangeable, and the two most commonly confused pairs will cost you real visibility if you get them backwards.

Every row verified against the operator’s own documentation · 2026-08-23

The mistake that costs the most

Blocking GPTBot opts you out of OpenAI’s model training. It does not remove you from ChatGPT. Blocking OAI-SearchBot does. People routinely block both, believing they are doing one thing, and disappear from ChatGPT’s answers as a side effect. The same split exists at Anthropic (ClaudeBot vs Claude-SearchBot) and at Perplexity, whose PerplexityBot is search-only and is not used for foundation model training at all.

The fear that is unfounded

Google states plainly that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.” It is also a robots.txt control token rather than a user-agent — you will never see it in your access logs. Opting out of Gemini training carries no documented SEO cost.

What the categories mean

Model training: Collects content that may be used to train foundation models. Blocking is an opt-out of training, and usually nothing else.
Search index: Builds the index the assistant searches when answering. Blocking removes you from that product’s answers entirely.
User-initiated fetch: Fetches a page because a person asked the assistant about it right now. Operators generally state that robots.txt may not apply, because the request is user-initiated.
Ad verification: Checks landing pages for advertising policy compliance.
General / unspecified: Fetches public content for purposes the operator does not tie to a specific product.

All documented crawlers

robots.txt token	Operator	Purpose	What blocking costs you
GPTBot	OpenAI	Model training	Opts your content out of training OpenAI’s foundation models. Does not remove you from ChatGPT search. Operator docs
OAI-SearchBot	OpenAI	Search index	Removes you from ChatGPT’s search results. This is the one that costs you visibility. Operator docs
ChatGPT-User may ignore robots	OpenAI	User-initiated fetch	Would stop ChatGPT fetching your page when a user explicitly asks about it. OpenAI notes robots.txt may not apply to user-initiated actions. Operator docs
OAI-AdsBot	OpenAI	Ad verification	Prevents safety validation of ad landing pages. Relevant only if you advertise through OpenAI. Operator docs
ClaudeBot	Anthropic	Model training	Opts your content out of contributing to Claude’s training data. Operator docs
Claude-SearchBot	Anthropic	Search index	Degrades or removes your presence in Claude’s search results. Operator docs
Claude-User may ignore robots	Anthropic	User-initiated fetch	Would stop Claude fetching your page when a user asks about it directly. Operator docs
Google-Extended token only	Google	Model training	Opts out of Gemini training and of grounding. Google states explicitly that it does not affect inclusion in Google Search and is not a ranking signal. Operator docs
GoogleOther	Google	General / unspecified	Blocks ad-hoc fetches by Google product teams. Not tied to any one product. Operator docs
PerplexityBot	Perplexity	Search index	Removes you from Perplexity’s search results. Perplexity states this bot is not used to crawl content for foundation models. Operator docs
Perplexity-User may ignore robots	Perplexity	User-initiated fetch	Would stop Perplexity visiting your page to answer a specific user question. Operator docs
CCBot	Common Crawl	Model training	Keeps you out of the Common Crawl corpus, which many organisations use as a training dataset. Also removes you from a widely used public research archive. Operator docs
Applebot-Extended token only	Apple	Model training	Opts out of Apple foundation model training without affecting Applebot, which powers Siri and Spotlight. Operator docs
meta-externalagent	Meta	Model training	Opts out of Meta’s AI training collection. Operator docs
Amazonbot	Amazon	Search index	Affects Alexa answers and Amazon search surfaces. Operator docs
Bytespider	ByteDance	Model training	Opts out of ByteDance model training. Frequently disallowed by publishers. Operator docs

Four things worth knowing

User-initiated fetches are largely outside your control

ChatGPT-User, Claude-User and Perplexity-User fetch a page because a person asked about it. OpenAI notes that “because these actions are initiated by a user, robots.txt rules may not apply.” Disallowing them in robots-ai.txt is mostly a statement of preference, not an enforcement mechanism — and blocking them means a user who explicitly wants your page gets a worse answer about it.

Some tokens are controls, not crawlers

Google-Extended and Applebot-Extended never appear in your logs. They are opt-out switches read from robots.txt while the fetch itself is performed by Googlebot or Applebot. This is why Applebot-Extended lets you opt out of Apple’s model training without breaking Siri and Spotlight.

Crawler rules are not the only lever on Google's AI surfaces

Google-Extended decides whether your content may be used for Gemini training and grounding. Separately, Google Search has a preferred sources feature: readers can mark a site as preferred, after which its content is more likely to surface in Top Stories and carries a “preferred” badge in AI Mode and AI Overviews. There is no file or markup to publish — eligibility is domain or subdomain level, and the only implementation work is an optional button or deeplink prompting readers to opt in. It is aimed at publishers appearing in Top Stories, so it will do nothing for most businesses. Worth knowing it exists; not worth confusing with an AI Discovery File.

This list will go out of date

Operators add, rename and retire crawlers without much ceremony. Each row above links to the source we checked it against — if you are making a decision that matters, click through and confirm. Tell us when something has moved and we will correct it.

Every row in this table is compiled from the operator’s own published documentation, linked in each row — not from any third-party list. It exists to make robots-ai.txt writable with intent. For the file’s definition, see its specification .