Make staging inaccessible; noindex is a second layer
The dependable way to keep a staging site out of public discovery is to require authentication or otherwise prevent public access. noindex is useful defense in depth for pages that crawlers can fetch, but it is not access control. Google’s robots meta guidance states that a crawler must be able to access a page to read and follow its indexing rules. A public robots.txt disallow is likewise a preference for compliant crawlers, not a lock.
Create an environment checklist: unique hostname, authentication at the edge or origin, production credentials excluded, no public sitemap or production canonical leakage, and a deliberate response for unauthenticated requests. Add a default X-Robots-Tag: noindex, nofollow or page meta directive only if it does not interfere with the authentication design. Record who can disable each layer and how the rule is prevented from shipping to the public host.
Test the boundary with a realistic deployment
Suppose preview.example.com is a copy of production for a redesign. It has a public 200 home page, a robots disallow, and staging URLs in its generated sitemap. A crawler that honors robots may avoid fetching it, but the environment is still visible to anyone who learns the address, and its noindex tag might never be read. Put the preview behind sign-in, ensure unauthenticated requests receive an appropriate protected response, remove its public sitemap exposure, and configure noindex as a safety net. Then confirm www.example.com has not inherited that header.
Use a fresh, unauthenticated request to inspect the preview root, a deep route, assets that contain sensitive content, robots.txt, sitemap.xml, headers, and canonical links. Test the production canonical host too. The key regression is often a shared CDN rule that adds noindex to every deployment. HTTP status codes for crawlers helps assess protected responses.
Do not confuse exclusion with secrecy
Do not put secrets in preview pages, source maps, or robots comments. Do not rely on an unguessable hostname. Do not copy a production sitemap into staging without review. Do not use a staging robots block as the only privacy measure. If a public staging URL was already exposed, assess it as an incident: fix access, remove sensitive material, and use the appropriate search removal process if needed.
The AI search readiness checker can inspect a URL you provide but should not be used to test an environment carrying private material. Its result is raw-fetch evidence, not a secrecy guarantee.
Make production and preview visibly different
Use environment-specific response checks rather than assuming a deployment variable is correct. Confirm unauthenticated preview access is denied, then confirm a production public page does not carry preview-only access or noindex headers. This reciprocal test catches the two expensive mistakes: exposed drafts and accidentally excluded live pages.
FAQ
Is noindex enough for a staging site?
No. Require access control first; use noindex only as an additional crawler instruction.
Should staging robots.txt disallow everything?
It can express that policy, but it neither authenticates users nor replaces a reviewed noindex/access setup.