A directive does not secure a resource
The Robots Exclusion Protocol asks compliant crawlers to follow site preferences. It is not an authorization system. A client can still request a disallowed URL; the server decides whether to return its content.
Use authentication for private material. Listing a sensitive path in a public robots file may reveal that the path exists.
Three questions, three kinds of evidence
Policy: Does a matching robots group allow the URL? Inspect the file and apply matching rules.
Generic HTTP observation: What response did one client receive? Record status, request identity, time and any redirect. This says nothing certain about another network’s treatment.
Verified crawler activity: Did a request originate from an operator’s independently verified infrastructure? This requires authoritative identity checks and logs, not just a familiar User-Agent string.
Our AI crawler checker provides policy interpretation and generic raw-fetch observations. It does not call them verified crawler visits.
Avoid accidental policy changes
Longer matching paths take precedence. Equal-specificity Allow and Disallow matches resolve to Allow. Repeated matching groups are combined. A bot-specific group should contain the restrictions you want that bot to observe; do not assume wildcard restrictions are inherited.
Before editing, save the existing file and identify who depends on it. Test representative public and excluded paths. After publishing, verify that the correct file is served on every relevant hostname.
Review enforcement separately
If you need access enforcement, review origin authentication and applicable CDN controls. Preserve access for legitimate users and required services. Do not broadly disable protections merely to make a diagnostic score rise.
The methodology explains the limits of our requests and evidence.