Overview of OpenAI crawlers
OpenAI
Reviewed 2026-09-08 · crawler policy
Read OpenAI documentation ↗Evidence, not assumptions
Official sources for AI crawler and website checks
Operator documentation defines crawler purpose. Standards define protocol behavior. Our recommendations remain distinct.
Anthropic crawlers and site controls
Anthropic
Reviewed 2026-09-08 · crawler policy
Read Anthropic documentation ↗Perplexity crawlers
Perplexity
Reviewed 2026-09-08 · crawler policy
Read Perplexity documentation ↗Common crawlers
Reviewed 2026-09-08 · crawler policy
Read Google documentation ↗Optimizing for generative AI features
Reviewed 2026-09-08 · AI search guidance
Read Google documentation ↗Bing crawler policy and robots testing
Microsoft
Reviewed 2026-09-08 · crawler policy
Read Microsoft documentation ↗About Applebot
Apple
Reviewed 2026-09-08 · crawler policy
Read Apple documentation ↗CCBot
Common Crawl
Reviewed 2026-09-08 · crawler policy
Read Common Crawl documentation ↗About Amazonbot
Amazon
Reviewed 2026-09-08 · crawler policy
Read Amazon documentation ↗DuckAssistBot
DuckDuckGo
Reviewed 2026-09-08 · crawler policy
Read DuckDuckGo documentation ↗RFC 9309: Robots Exclusion Protocol
IETF
Reviewed 2026-09-08 · robots specification
Read IETF documentation ↗Sitemap protocol
Sitemaps.org
Reviewed 2026-09-08 · sitemap specification
Read Sitemaps.org documentation ↗Getting started
Schema.org
Reviewed 2026-09-08 · structured data
Read Schema.org documentation ↗Robots meta and X-Robots-Tag
Reviewed 2026-09-08 · indexability
Read Google documentation ↗WCAG 2.2
W3C
Reviewed 2026-09-08 · accessibility
Read W3C documentation ↗Freshness policy
Mutable crawler records are flagged after thirty days without review. Automation checks source availability and raises a review issue; it never silently changes facts. A successful HTTP response does not establish factual verification.