Find the layer that denied the request
An Allow in robots.txt does not override a WAF challenge, rate limit, origin authorization check, or application error. Robots communicates crawler preferences; the Robots Exclusion Protocol also warns that it is not a substitute for access security. Troubleshooting should identify the layer that produced the response before anyone weakens a rule.
Start with one failed request or a narrow time window. Obtain the edge request ID, UTC time, hostname, normalized path, trusted client IP, full User-Agent, response status, and WAF event. If the report came from an external test, reproduce only the public response; do not call it a genuine provider request.
Trace the response from edge to origin
Work through the control path in order:
- Robots: calculate the effective rule for the claimed product token and exact path. This says whether the crawler is invited, not whether the request passed enforcement.
- CDN connectivity: confirm DNS, TLS, redirect behavior, and the final hostname. A redirect can move the request onto a host with different WAF and robots policies.
- Bot management: inspect the exact rule ID and action. Record whether the WAF blocked, challenged, rate-limited, or merely logged the request.
- Custom rules: check path, country, method, header, and IP-list conditions. Rule priority matters when a broad block runs before a narrow allow.
- Origin: compare the edge status with the upstream status. Authentication middleware, load shedding, or an application exception may produce the denial after the WAF permits it.
- Content: after a
200, confirm the body is the intended page rather than a challenge shell or soft error.
The AI Crawler Checker can reproduce a bounded public fetch, and the robots.txt AI Checker can explain a path match. Neither has access to private WAF events, so use the matching request ID in your own provider console.
Verify identities before creating exceptions
Never allow traffic solely because the User-Agent contains a familiar bot name. Google supports reverse-plus-forward DNS and crawler-category IP lists in its verification guide. Perplexity publishes separate current ranges for PerplexityBot and Perplexity-User and recommends pairing the correct range with its matching User-Agent in its WAF guidance. Other providers can have different identities and evidence.
An exception should include the verified operator condition, public hostname, required route class, safe methods, and only the WAF action causing the fault. Keep login, checkout, uploads, search, and administrative APIs under their existing protection unless the business requirement explicitly includes them. Add an expiry or review date when the condition depends on an external range list.
Explicitly hypothetical incident
Assume a fictional documentation site receives a complaint that Perplexity cannot fetch /guides/install. Robots permits PerplexityBot. An external request returns 403, but that alone says nothing about the actual crawler.
The team finds a real edge event whose User-Agent claims PerplexityBot. The trusted visitor IP matches the current PerplexityBot list, and the denial came from a generic browser-challenge rule. The origin was never contacted. They add an exception requiring both the current PerplexityBot IP set and token, limited to GET and HEAD on /guides/. The account and /api/search routes remain excluded.
Retesting through the edge reaches the origin and returns the guide with 200; a request to /account still receives 401. The incident closes as a corrected WAF path for verified crawler traffic. It does not claim indexing or citation. If the trusted address had been missing, the correct result would have remained unknown while logging was repaired.
FAQ
Does an allowed robots rule bypass a WAF block?
No. Robots policy and HTTP enforcement are separate controls.
Is a broad “allow known bots” rule safe?
No. Verify the specific provider identity and scope the exception to the required public routes and methods.