Logs answer activity questions, not citation questions
AI crawler log analysis can show which requests reached your infrastructure, how the edge handled them, which routes consumed resources, and which identities were verified. It cannot show whether an AI product indexed a page, used its content in an answer, or displayed a citation. Those later events need different evidence.
Start with a bounded question such as “What claimed AI crawlers requested public documentation last week, and what share received 2xx responses?” Define the time window and host before exporting anything. This keeps the analysis reproducible and reduces unnecessary retention of sensitive URLs.
Create one privacy-conscious event table
Export only fields needed to answer the question: UTC timestamp, trusted client IP, full User-Agent, hostname, method, route class, status, bytes, cache outcome, edge action, and request duration. Replace raw query strings and customer identifiers with a coarse route such as /search or /account/:id. Do not copy response bodies into the dataset.
Add classification fields rather than rewriting the original evidence:
claimed_identity: token parsed from the User-Agent, such as OAI-SearchBot or GPTBot.purpose: search discovery, training-related collection, user-triggered retrieval, or unknown, based on current provider documentation.verification_result: verified, unknown, or not checked.verification_evidence: DNS method or range-list URL and retrieval time.route_cost: cached public page, origin-rendered page, dynamic search, API, or protected route.
OpenAI’s bot documentation distinguishes OAI-SearchBot, GPTBot, and ChatGPT-User. Preserve those identities instead of reporting one “OpenAI bot” total. Apply the same principle whenever another provider publishes separate search, training, and user agents.
Verify before comparing behavior
Treat the User-Agent as a claim. For Google requests, the current official verification guide supports reverse-then-forward DNS and category-specific published IP ranges. Other providers may publish their own ranges or procedures. The crawler verification guide covers trusted proxy headers, suffix checks, and unknown results.
Verify a stratified sample across identities, time periods, and route classes. If the verified share varies widely, report verified and unverified counts separately; do not multiply a small sample into a precise sitewide number. DNS failure or missing proxy provenance is unknown, not automatically hostile.
Aggregate after classification. Useful tables include requests by identity and status family, unique verified IPs by day, cache hit rate by route class, 95th-percentile duration, and edge actions. Compare cached articles with expensive filters or search routes. A request count without cost can make a harmless cached crawl look more urgent than a smaller burst of origin-heavy work.
Explicitly hypothetical weekly analysis
Suppose a fictional news archive exports seven days of edge events. It finds 48,000 requests whose User-Agent contains an AI-related token. After stripping query values, it groups routes and verifies a sample of 600 source addresses using current provider methods.
The hypothetical result is uneven: most verified search-crawler requests hit cached articles with a 96% cache-hit rate, while a large unverified group repeatedly requests /search and produces slow origin queries. The analyst reports the two populations separately. Operations adds a general unauthenticated rate limit to /search, because route cost is established but operator identity is not. Public article policy stays unchanged.
The report also notes that 12% of sampled claimed requests could not be verified because an old CDN export lacked the trusted visitor-IP field. It recommends fixing future log configuration; it does not recast those rows as verified or malicious. Nothing in the analysis is described as a citation, ranking, or referral result.
FAQ
Should I merge claimed and verified crawler counts?
No. Keep both so spoofed or unverifiable requests do not become first-party crawler facts.
Can access logs tell me which pages an assistant cited?
No. Logs show HTTP requests. Citation tracking requires a separately defined observation method.