AI Crawl Score
Menu

Rate-limiting AI crawlers

Rate-limit expensive routes using trusted identity and measured capacity.

Sources reviewed 2026-09-10

Limit the expensive work, not the bot label

Rate-limit AI crawlers when measured request patterns threaten capacity or unfairly consume an expensive route. Start with route cost and service limits. A User-Agent is easy to spoof, and a verified crawler still should not receive unlimited access.

Separate cached documents from endpoints that trigger database searches, filters, renders, exports, or third-party calls. A thousand cache hits may cost less than ten concurrent report generations. If the expensive operation should not be publicly crawlable at all, fix navigation, authentication, or robots policy first; rate limiting is then a resilience layer rather than the only control.

Establish a capacity baseline

Use recent edge and origin data to record request rate, concurrency, cache misses, upstream latency, error rate, and compute or database load for each route class. Choose a limit that preserves human use and origin headroom during normal bursts. Document the time window, key, threshold, burst allowance, and action so an operator can reproduce the decision.

A practical sequence is:

  1. Group URLs into static public pages, cached dynamic pages, uncached search/filter routes, authenticated APIs, and write operations.
  2. Remove query values and account identifiers from analysis data while retaining the route class.
  3. Confirm which layer can enforce a limit before expensive work begins. Edge enforcement usually protects the origin better than an application response issued after a database query.
  4. Pick a trustworthy key: verified source network, authenticated account, API credential, or a combination of source IP and route. Do not key privileged treatment on User-Agent alone.
  5. Return 429 Too Many Requests when a client exceeds the limit. RFC 6585 defines 429 and says a response may include Retry-After; HTTP Semantics defines that header as either a delay in seconds or an HTTP date in RFC 9110.
  6. Test behavior below, at, and above the threshold, including IPv6, multiple edge locations, and recovery after the window expires.

The AI crawler log analysis guide provides a bounded method for separating route cost from identity claims. Public-response tools cannot configure or continuously monitor your infrastructure.

Prefer shared limits until evidence supports an exception

For an expensive unauthenticated route, a general client limit is often safer and simpler than one rule per claimed crawler. Add a provider-specific allowance only when its identity can be verified, the route is genuinely intended for that crawler, and the shared limit demonstrably blocks useful access. Keep the exception narrower than the original rule.

Log the policy outcome, not sensitive request contents: rule ID, normalized route, status, retry interval, trusted identity result, and duration. Track whether clients retry immediately, back off, or rotate addresses. Address rotation may require a broader network or route control, but avoid assuming coordination from a small sample.

Review the interaction with caches and conditional requests. A limit on every request may punish clients that are revalidating unchanged documents, while a limit applied only after a cache miss may leave an expensive origin path exposed. Test If-None-Match and If-Modified-Since behavior, and make sure a 304 response is not counted as a costly render when it is not. Give humans a clear retry message and avoid exposing internal capacity numbers in a public error body.

When a provider publishes a crawl schedule or retry guidance, treat it as context rather than a contract. Your edge still needs a safe default when identity is unknown. Keep a short incident note explaining why the threshold changed, what workload was measured, and when to revisit it after a deployment or database migration.

Explicitly hypothetical marketplace rollout

Suppose a fictional marketplace observes origin latency spikes on /products/search?filters=.... Static product pages are cached, but every filtered request runs several database queries. During one hypothetical hour, claimed crawler traffic produces 18 requests per second to search while ordinary peak traffic needs 12; the origin becomes unstable above 24 combined.

The team applies an edge token bucket to unauthenticated /products/search, allowing a short burst and then returning 429 with Retry-After: 30. Cached /products/:slug pages remain outside that strict limit. Authenticated merchant tools keep their existing account quotas. A verified search crawler is later granted a modest route-specific allowance after logs show that the shared threshold repeatedly interrupts its normal sequence.

Load testing confirms search stays within its origin budget and recovers after the window. The recorded result is capacity protection under the tested workload. It does not claim that the crawler will obey every retry hint, index more products, or change citations.

FAQ

Should verified AI crawlers bypass all limits?

No. Verification establishes identity, not unlimited capacity or authorization.

Does a 429 response change robots.txt policy?

No. It reports current HTTP rate enforcement; crawler preference remains a separate decision.

Primary sources