01 · The foundations
Three processes to keep apart
- 01 Crawling collects pages
- A crawler requests web content. That collection may support a search index, a training dataset or another product. Its documented purpose matters; a request in your logs does not establish how the content was later used.
- 02 Training changes the model
- Training adjusts a model’s internal parameters using selected data. It can then answer using what it learned, without opening those original pages. A training cutoff describes a boundary for learned knowledge; it does not set the age of information a search tool can retrieve.
- 03 Retrieval supplies information for an answer
- A product can search an index or fetch a page while answering. It passes relevant material to the model as context, often called grounding or retrieval-augmented generation (RAG). That can bring newer information into the answer without retraining the model.
For example, xAI describes pre-training as updating model weights, separately from Grok’s web search. Google describes grounding as retrieving information from its search index. Those are different routes by which information can influence an answer.
“Live search” does not mean every page was freshly fetched
A search during a conversation can return material from an existing index or cache. OpenAI’s Responses API, for example, explicitly offers live access and a cache-only mode. This API setting does not describe every ChatGPT conversation. Check the cited page’s own date and contents.
Training data reaches models through several routes
Web crawling is one route. AWS also describes licensed and proprietary data, open datasets and synthetic data, with human and automated curation. Common Crawl makes a web dataset available to others. These disclosures are examples of data supply; they do not identify every model’s inputs.
Do not infer that a model trained on your page because it names your business, or that blocking future collection removes earlier learning. A citation is evidence of a source presented with an answer, not a training-data receipt.
02 · Compare providers
The product determines how search is used
For Reddit, LinkedIn, Facebook, Instagram and TikTok, see our social platform access reference. It includes dated crawler rules, API partnerships and the limits of claims about ChatGPT’s Bing integration.
A model and the application around it are different things. Compare the specific product, search setting and answer. The same model can be used with different tools and sources.
| Experience | Documented route to web information |
|---|
| Google Search | AI features use Google’s Search index and ranking systems. |
|---|
| Bing and Copilot | Bing search infrastructure; Copilot Chat can send queries to Bing. |
|---|
| ChatGPT / OpenAI | Search crawling and user-requested page visits have distinct roles. |
|---|
| Claude / Anthropic | When enabled, web search and direct URL fetch bring sources into a chat. |
|---|
| Grok / xAI | Can search the public web and public X posts. |
|---|
| Perplexity | Search indexing and user-triggered retrieval have separate agents. |
|---|
Google Search
Google’s generative Search features retrieve from its core index and may issue related searches to explore a question. Its SEO guidance applies to these features.
Googlebot controls Search crawling. Google-Extended covers future Gemini training and specified Gemini/Vertex grounding uses; Google says it does not affect Search inclusion or ranking. Check which product you intend to control.
AI inclusion: Search Console’s Settings → Search generative AI controls inclusion in AI Overviews, AI Mode and generative AI features in Discover. Inclusion is the default; child properties inherit a parent’s choice unless overridden. Check this separately from crawl rules. It does not control AI training or inclusion in other parts of Search.
Bing and Microsoft Copilot
Bing’s guidelines connect SEO and GEO through shared crawling, indexing and ranking foundations. Microsoft’s documentation for Copilot Chat and agents describes sending a short query to Bing when web search is enabled. Organisation settings can control access.
Controls: Bing uses bingbot for crawling. Its page-level directives also affect AI use: noarchive restricts Copilot use and Microsoft foundation-model training; nocache has different, narrower effects. Read the exact combinations before applying them.
ChatGPT and OpenAI
OAI-SearchBot supports ChatGPT search visibility; GPTBot collects potential foundation-model training content. OpenAI documents independent choices for these uses. ChatGPT-User handles certain user-requested visits, where robots.txt may not apply.
Practical difference: allowing search while restricting training is a documented option. These crawler roles do not reveal a complete list of search suppliers or the sources used in a particular answer. The Responses API’s search settings should be assessed separately from ChatGPT.
Claude and Anthropic
Claude can invoke web search for current information when enabled, return citations, and fetch a supplied URL directly. Anthropic separates ClaudeBot for potential training content, Claude-SearchBot for search indexing and Claude-User for user requests.
Controls: Anthropic documents robots.txt rules for all three. Its web-search help page identifies Bing for image results; that specific statement does not establish the provider of every text search.
Grok and xAI
Grok’s consumer documentation distinguishes pre-training on public information from searching the web and public X posts. A current X result and a fact learned during training can therefore enter an answer through different routes.
The cited FAQ does not establish a website crawler token or a complete search-index supplier list, so it cannot support a Grok robots.txt rule. Account controls for conversation training are separate from website crawl access.
Perplexity
PerplexityBot supports search results; Perplexity-User can fetch pages for a user’s question. Perplexity says neither collects content for foundation-model training. It says the user fetcher generally ignores robots.txt.
This policy describes the two agents’ purposes; the training history of models available through Perplexity is a separate question.
03 · Beyond chat assistants
Apple and Amazon also separate uses
Apple: Applebot supplies search and other features. Applebot-Extended is a training-use control, not another crawler; blocking it still permits search inclusion. Apple also documents separate controls for content used as context in AI answers.
Amazon: Amazonbot can collect content for product development and training. Amzn-SearchBot supports search experiences such as Alexa, while Amzn-User can fetch information for a customer request. Amazon says these latter two agents do not collect for generative-model training.
04 · Put this to use
Choose access, then measure the outcome
Use the quick guide’s access checks to apply the provider controls above, then follow its measurement routine. Access, indexing, citations and useful visits answer different questions.
For nearby services or a named place, the local search reference explains searcher location, service areas and how to compare results in context.
05 · Keep the evidence dated
What our timestamps mean
Fetched records when we retrieved the documentation for this review. Reviewed records when we checked the summary against it. Both use UTC. A publisher’s own date is shown separately where available; it may be relative or absent.
These are documentation checks, not tests of crawler compliance or guarantees that a service still behaves this way. Dates stay fixed until another source review; rebuilding this website does not refresh them. Reopen the linked documentation before changing access rules.
We identify product and API scope where it matters. Undocumented suppliers, training inputs and controls remain unknown. If a source changes or a summary needs correcting, let us know.
