Decide what you want to permit
OpenAI documents OAI-SearchBot as a search crawler and GPTBot as training-related collection. ChatGPT-User performs user-triggered actions; robots rules may not apply to those requests. These are separate decisions, not three interchangeable labels.
If you want search discovery while excluding training collection, a starting policy is:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /Preserve existing path restrictions
Do not paste this over an existing policy without review. Suppose your wildcard group disallows /members/. A specific OAI-SearchBot group is considered separately. Copy applicable private-path exclusions into that group:
User-agent: OAI-SearchBot
Allow: /
Disallow: /members/This is a preference for compliant crawlers, not a substitute for authentication on private pages.
Verify the layers independently
First publish the intended robots file at the host root. Then run the policy checker and inspect the matched rule. Finally review your CDN and origin controls. A readable robots file does not establish whether an operator’s verified infrastructure can retrieve a page.
A generic request with a crawler’s name in the User-Agent header is not verified crawler traffic. Use official identity-verification guidance and your own logs when investigating actual visits.
Interpret the score correctly
AI Crawl Score does not penalize GPTBot restrictions. Search policy, HTTP behavior, indexability and available page content are separate evidence. None guarantees a ChatGPT citation.
Read the OAI-SearchBot field guide and the GPTBot field guide before changing a policy you inherited.