They control different outcomes
Googlebot controls crawling for Google Search and related Search surfaces. Google-Extended is a separate robots.txt product token that lets publishers manage whether content Google crawls may be used for future Gemini model training and for grounding in Gemini Apps and Vertex AI. Google states that Google-Extended does not affect inclusion or ranking in Google Search in its common crawlers documentation.
Google-Extended also has no separate HTTP User-Agent string. Google says crawling uses existing Google user agents, while the Google-Extended token communicates the publisher’s usage preference. Therefore, you cannot identify a “Google-Extended visit” in a log by searching for that token.
Write two independent policy decisions
Begin with plain language: “Keep these public pages eligible for Google Search” and “Allow or disallow these pages for Google-Extended uses.” Translate each sentence to its own named group. Do not block Googlebot as a shortcut for the second decision.
Review the change in this order:
- Fetch
/robots.txtfrom every hostname and retain the deployed text. A documentation subdomain may serve a different policy from the main site. - Test a matrix of
GooglebotandGoogle-Extendedagainst a small set of paths. Include the home page, one key article, a private-looking path, and any exception created withAllow. - Apply longest-match behavior when
AllowandDisallowoverlap. Do not judge the file from line order alone. - For Google Search readiness, separately inspect final status, canonical, robots meta or
X-Robots-Tag, snippet eligibility, and meaningful page content. A permissive Googlebot rule cannot overridenoindex, a login wall, or a broken response. - Deploy, fetch the live file again, and record the exact version and time. CDN caches can leave an old robots file active after the repository changes.
The robots.txt AI Checker can calculate the effective policy for a URL. For a broader but still bounded page check, use the AI search readiness checker.
Read results without inventing a crawler
If server logs show Googlebot, verify that identity using Google’s current documented DNS or IP-range method before creating a WAF exception. The request shows Google crawling activity; it still does not reveal whether Google-Extended permitted a downstream use. The Extended control is evaluated from robots policy, not from a special visitor label.
Changing Google-Extended also does not promise a Gemini citation. Policy permission, Google Search indexing, grounding selection, answer presentation, and referral traffic are separate events. Keep each evidence type in its own report.
Hypothetical documentation change
Suppose a fictional database vendor, JuniperDB, wants its manuals in Google Search but chooses to disallow Google-Extended for /docs/enterprise/. An editor adds a named Google-Extended rule while leaving Googlebot unchanged.
The release audit finds that an older wildcard Disallow: /docs/ does not control Googlebot because a separate Googlebot group permits /docs/public/; however, /docs/enterprise/setup contains noindex from a staging template. The team removes the accidental meta directive from the public manual and confirms its canonical is self-referential. It then retests the two tokens: Googlebot is allowed on the public setup guide, while Google-Extended is disallowed on the enterprise path.
The report records two facts: a Search indexability defect was repaired, and an Extended preference was deployed. It does not merge them into an “AI visibility score” or claim a ranking change.
FAQ
Does disallowing Google-Extended remove a page from Google Search?
No. Google says Google-Extended does not affect Search inclusion or ranking.
Can logs show a Google-Extended crawler?
No separate one is documented. Google-Extended is a robots product token, not its own HTTP User-Agent.