Match the most specific rule, not the rule that looks strongest
Robots patterns are path rules, not folders with inheritance. Under RFC 9309, a crawler chooses the most specific matching rule; when equally specific allow and disallow rules match, the allow rule wins. Google documents compatible interpretation details in its robots guide. Treat wildcards as a compact way to express a tested policy, not as punctuation to scatter through a file.
* matches a sequence of characters and $ anchors a rule at the end of a URL path. A plain Disallow: /search can match more than /search/; a trailing $ is useful when the exact ending matters. Query-string behavior is easy to misread because robots matching uses the URL path and query components as supplied to the crawler, so verify the concrete URLs your application emits.
Build rules from examples first
Before writing a pattern, list five URLs it must allow and five it must disallow. Include trailing slashes, pagination, parameters, assets, and alternate hostnames. Put a short comment above a non-obvious rule explaining the product reason. Then run each case through the robots.txt AI checker and inspect the matched path rather than trusting a green or red label.
For example, a shop wants to conserve crawl requests on internal result pages but retain category pages. Disallow: /search also blocks /searchable-guides/ if that route exists. Disallow: /search? is narrower but may miss a route with a slash. The team should first decide its stable search URL shape, then test https://shop.example/search?q=boots, /search/, and /searchable-guides/. A rule only earns its place after those results match the intended inventory.
Keep crawler groups complete
Specific user-agent groups are evaluated for that agent. Do not assume a bot-specific group automatically inherits every wildcard instruction. Repeated matching groups may be combined, which makes separately maintained snippets surprisingly risky. Consolidate each important crawler’s policy in one reviewed file and retest after a CMS, CDN, or localization change.
The AI crawler checker helps distinguish a published policy from a generic HTTP observation. It cannot establish that a named operator fetched the page. For the privacy and security boundary, read robots policy is not enforcement.
Confirm the served file
Retrieve /robots.txt from the exact public hostname and protocol after publication. A preview can serve a different file, a CDN can cache an old copy, and a redirect can put policy on another host. Keep representative URL tests in a release checklist so later edits preserve the purpose of each wildcard rather than merely its syntax.
FAQ
Document one permitted and one prohibited URL beside each rule. Future editors can then change a path confidently and recognize when a seemingly harmless wildcard has crossed a boundary.
Does a longer rule always win?
The most specific matching rule generally wins; equal-specificity Allow and Disallow resolve to Allow under the standard.
Should I block URL parameters with a wildcard?
Only after testing actual emitted URLs and deciding whether those variations should be crawled, canonicalized, redirected, or kept accessible.