Decide for three Claude identities
Claude crawler policy should be written as three decisions, because Anthropic assigns different jobs to Claude-SearchBot, ClaudeBot, and Claude-User. Its crawler help article describes Claude-SearchBot as search-oriented, ClaudeBot as collection that may contribute to model training, and Claude-User as retrieval initiated by a person using Claude. Allowing one does not require allowing all three.
This separation lets a publisher support search discovery, decline training collection, and decide independently whether user-requested fetches should reach public content.
Turn the policy into explicit routes
List content classes before editing robots.txt: public articles, public documentation, search results, account pages, staging URLs, downloads, and APIs. Give every Claude identity an intended outcome for each class. Authentication remains mandatory for private classes regardless of robots policy.
Then check the deployed behavior:
- Retrieve
/robots.txtfrom every affected hostname. Robots policy is scoped to the service where the file is served, so the main site and documentation subdomain need separate review. - Evaluate the named Claude group and the wildcard fallback for representative URLs. Anthropic says its bots honor robots directives and also documents a non-standard
Crawl-delayextension; treat that extension as provider-specific rather than a universal robots feature. - Request an allowed page and follow its redirect chain. Record final status, canonical target,
X-Robots-Tag, HTML robots meta, and whether the answer is present in raw HTML. - Review WAF logs for challenge, block, or rate-limit actions. Robots permission cannot bypass HTTP enforcement.
- Save the exact policy, tested paths, time, and expected owner. Recheck after CDN or publishing changes.
Use the robots.txt AI Checker for path-level rule interpretation. The AI Crawler Checker can inspect a public response, but its request is not evidence that Anthropic reached the URL.
Avoid accidental coupling
A broad User-agent: * rule may affect a Claude identity only when no more specific matching group applies. Do not assume a wildcard statement and a named statement combine the way a casual reading suggests; test the effective rule for the actual token and path.
Keep operational enforcement equally narrow. If a dynamic search endpoint is expensive, protect or rate-limit that endpoint for unauthenticated traffic instead of opening it because a crawler is allowed elsewhere. If Claude traffic is suspected in logs, keep “claimed by User-Agent” distinct from “verified using current provider evidence.” A copied name in an HTTP header is easy to spoof.
Hypothetical publisher rollout
Imagine a fictional medical publisher, Cedar Health Library. It wants Claude search discovery for peer-reviewed public explainers, opts out of ClaudeBot collection, and permits Claude-User only on public articles. It deploys named groups for all three and keeps /patient/ behind login.
During testing, /conditions/asthma is allowed for Claude-SearchBot but returns a JavaScript challenge from the WAF. The security team finds that a generic automation rule caused the challenge. It creates a route-limited exception using the strongest provider verification available, while keeping upload, account, and search routes under the original rule. A second request reaches the article with 200; the raw response includes its review date and citations.
The release record says the public article became reachable under the tested edge path. It does not claim Claude indexed or cited it. The team schedules a policy review when Anthropic changes its documentation, not continuous monitoring supplied by this site.
FAQ
Does disallowing ClaudeBot also disallow Claude-SearchBot?
No. Anthropic publishes them as separate product tokens, so state both policies explicitly.
Should robots.txt protect patient or account data?
No. Robots files are public preferences. Use authentication and authorization for private data.