Decide what you want to permit
OpenAI documents OAI-SearchBot as a search crawler and GPTBot as training-related collection. ChatGPT-User performs user-triggered actions; robots rules may not apply to those requests. These are separate decisions, not three interchangeable labels.
Confirmed fact: OpenAI says the OAI-SearchBot and GPTBot settings are independent. It publishes separate current IP lists for OAI-SearchBot, GPTBot, and ChatGPT-User.
If you want search discovery while excluding training collection, a starting policy is:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /Preserve existing path restrictions
Do not paste this over an existing policy without review. Suppose your wildcard group disallows /members/. A specific OAI-SearchBot group is considered separately. Copy applicable private-path exclusions into that group:
User-agent: OAI-SearchBot
Allow: /
Disallow: /members/This is a preference for compliant crawlers, not a substitute for authentication on private pages.
Verify the layers independently
First publish the intended robots file at the host root. Then run the policy checker and inspect the matched rule. Finally review your CDN and origin controls. A readable robots file does not establish whether an operator’s verified infrastructure can retrieve a page.
A generic request with a crawler’s name in the User-Agent header is not verified crawler traffic. Use official identity-verification guidance and your own logs when investigating actual visits.
Industry observation: WAF rules often classify unfamiliar automation before robots policy matters. A 403 in a manual test identifies an access problem, not its cause.
Recommendation: When a real request needs verification, require both the expected User-Agent token and membership in the matching current OpenAI IP list. Keep that exception limited to public routes and refresh the list instead of copying its addresses permanently.
Hypothesis: Removing an unintended WAF block may make an eligible page discoverable by ChatGPT Search. It does not prove that OpenAI will crawl, select, or cite the page.
Interpret the score correctly
AI Crawl Score does not penalize GPTBot restrictions. Search policy, HTTP behavior, indexability and available page content are separate evidence. None guarantees a ChatGPT citation.
Read the OAI-SearchBot field guide and the GPTBot field guide before changing a policy you inherited.