OAI-SearchBot
OpenAI · search
Search discovery for ChatGPT. Training preferences are controlled separately with GPTBot.
Robots token: OAI-SearchBot
Source reviewed
OpenAI crawler documentation ↗Read the field guide →The crawler registry
AI crawler directory
These records drive the policy matrix. Robots tokens are central; full User-Agent browser versions may change.
GPTBot
OpenAI · training
Training-related collection. Blocking this crawler does not itself block OAI-SearchBot.
Robots token: GPTBot
Source reviewed
OpenAI crawler documentation ↗Read the field guide →ChatGPT-User
OpenAI · assistant
User-initiated retrieval; robots.txt may not apply. This is not the ChatGPT search-index crawler.
Robots token: ChatGPT-User
Source reviewed
OpenAI crawler documentation ↗Claude-SearchBot
Anthropic · search
Collects content to improve search responses; separate from training collection.
Robots token: Claude-SearchBot
Source reviewed
Anthropic crawler documentation ↗Read the field guide →ClaudeBot
Anthropic · training
Collects material that may contribute to model training; an independent policy choice.
Robots token: ClaudeBot
Source reviewed
Anthropic crawler documentation ↗Claude-User
Anthropic · assistant
Retrieves pages in response to user requests. Anthropic documents robots.txt compliance.
Robots token: Claude-User
Source reviewed
Anthropic crawler documentation ↗PerplexityBot
Perplexity · search
Surfaces websites in Perplexity search. Not a foundation-model training crawler.
Robots token: PerplexityBot
Source reviewed
Perplexity crawler documentation ↗Read the field guide →Perplexity-User
Perplexity · assistant
User-requested retrieval generally ignores robots.txt. Policy here is not proof of enforcement.
Robots token: Perplexity-User
Source reviewed
Perplexity crawler documentation ↗Googlebot
Google · search
Google Search crawler. Google-Extended is a distinct use-control token.
Robots token: Googlebot
Source reviewed
Google crawler documentation ↗bingbot
Microsoft · search
Bing Search crawler. Read robots policy independently from verified crawler identity.
Robots token: bingbot
Source reviewed
Microsoft crawler documentation ↗Applebot
Apple · mixed
Powers Apple discovery and other documented uses. Applebot-Extended separately controls training use; Googlebot policy is the documented fallback.
Robots token: Applebot
Source reviewed
Apple crawler documentation ↗CCBot
Common Crawl · mixed
Builds a public web corpus usable for research and other downstream purposes. Blocking it is not a search-readiness failure.
Robots token: CCBot
Source reviewed
Common Crawl crawler documentation ↗Amazonbot
Amazon · mixed
Collects public content for Amazon services. Consult the operator for current use and control details.
Robots token: Amazonbot
Source reviewed
Amazon crawler documentation ↗Google-Extended
Google · training
Robots control token, not a separate HTTP crawler. Google says crawling uses existing Google user agents. The token covers Gemini training and grounding, and it does not affect Google Search inclusion or ranking. Evaluated as its own group, otherwise *.
Robots token: Google-Extended
Source reviewed
Google crawler documentation ↗Applebot-Extended
Apple · training
Robots control token, not a crawler. Apple says Applebot-Extended does not fetch pages; it limits use of content Applebot already crawled for foundation-model training. Evaluated as its own group, otherwise *. Applebot’s Googlebot fallback is separate and is not applied to this token.
Robots token: Applebot-Extended
Source reviewed
Apple crawler documentation ↗DuckAssistBot
DuckDuckGo · search
Retrieves information for DuckDuckGo AI-assisted answers.
Robots token: DuckAssistBot
Source reviewed
DuckDuckGo crawler documentation ↗