Set X-Robots-Tag on the PDF response
PDF files cannot contain an HTML <meta name="robots"> element, so use an X-Robots-Tag HTTP response header when you need an indexing or serving directive for one. Google explicitly recommends the header for non-HTML resources such as PDFs, videos, and images in its robots meta specification. The crawler still needs access to the file to read that instruction.
For a PDF that should remain downloadable but not appear in search results, send X-Robots-Tag: noindex with its successful response. Set the header at the layer that serves the file: origin, object-storage rule, CDN response transform, or application route. Avoid attempting to add it through a page that merely links to the PDF; the file response is what matters.
Verify the final response, not the configuration screen
Request the exact public PDF URL and inspect response headers after redirects. Confirm the final status is the expected successful status, its Content-Type identifies the document, and the header survives the CDN. Test the canonical hostname and a cache-bypassing deployment only when your platform supports it safely. Then use the AI search readiness checker for the surrounding site, while remembering its page scan is not a document-indexing confirmation.
Imagine a university replaces /downloads/2024-admissions.pdf with a draft that contains unapproved fees. It leaves the old file available for historic links but uses X-Robots-Tag: noindex on that one object while a corrected replacement is prepared. Adding a noindex tag to /admissions/ does nothing to the old PDF. Blocking the PDF in robots.txt could also prevent a crawler from seeing the header. If the draft itself must be private, remove public access and use authorization instead.
Choose directives with a clear purpose
noindex addresses result inclusion. Other supported rules can control snippets or previews, but only use them when a documented user or publisher need requires them. Google notes that conflicting robots rules resolve to the more restrictive instruction. Keep a small inventory of document paths and their headers, especially after moving files between a CMS and a CDN.
Common mistakes are returning the header only on the HTML download landing page, applying it to an entire storage bucket unintentionally, and assuming it hides a publicly reachable document. Review robots.txt vs noindex before combining controls. A generic request says what that response sent; it cannot prove how every crawler will act.
Assign ownership of the response
Marketing may own a landing page while infrastructure owns object headers. Give both teams the exact document URL and desired final response, then retain a small header check after storage migrations. This prevents a copied or renamed PDF from silently losing its instruction when a platform default replaces the prior metadata.
FAQ
Retest the exact final document URL after a cache purge, deploy, or file replacement. The file name, redirect target, and response header must remain aligned for the decision to hold.
Can I put a robots meta tag inside a PDF?
No. Use X-Robots-Tag on the HTTP response for a non-HTML resource.
Does noindex stop a PDF download?
No. It is an indexing directive. Change access controls if downloading must be restricted.