What does this token control?
Use Claude-SearchBot as the robots product token. Treat it separately from incidental browser/version text in a full User-Agent header. A token in a request does not establish the operator’s identity.
A policy example
User-agent: Claude-SearchBot Allow: /
This permits the search crawler. Before merging, copy applicable private-path restrictions from your wildcard group.
Common mistakes
ClaudeBot collects potential training material. Claude-SearchBot supports search. Claude-User retrieves pages at a user’s direction. Review each policy independently.
Test public evidence
Inspect the matched policy, then review your server and CDN controls. Our request identifies as AICrawlScoreBot and cannot establish treatment of verified operator infrastructure.
Check crawler policy → · Generate a snippet
Official reference
Anthropic crawler documentation. Use the source for full User-Agent examples and supported identity-verification methods.