Check explainer · v1
X-Robots-Tag: the noindex you cannot see in the HTML
Last reviewed
The header says noindex and the HTML says nothing. Anyone checking the page source sees a clean page — and the site quietly drops out of the index.
How this check works
Every fetched page’s X-Robots-Tag response header is read for noindex or none. A match fails the page with the header value quoted: the directive is equivalent to a meta robots noindex, but it lives at the HTTP layer — invisible to page-source views and to tools that only read HTML.
What does a failure look like?
A CDN or edge rule that stamps X-Robots-Tag: noindex on a whole environment — staging config promoted to production; an auth middleware that marks every response; a bot-protection layer that answers with noindex on suspicious traffic — links and sitemaps intact, indexing silently off.
How to fix it, step by step
- Find the layer that sets the header: edge worker, CDN rule, reverse proxy or application middleware.
- Scope the directive to the environments that need it — staging yes, production no — and redeploy.
- Re-check a sample URL with the single-page noindex check and confirm the header is gone.
What does a pass or fail look like?
Good
No page carries noindex or none in its X-Robots-Tag response header.
Fail
Every response ships X-Robots-Tag: noindex, nofollow from an edge rule — the HTML is clean, and the whole site is excluded.
Frequently asked questions
Why can page-source tools miss this entirely?
Because X-Robots-Tag is a response header, not page markup. Tools that fetch and parse HTML never see it, so a site can look perfectly indexable in every page-source view while the header excludes everything.
Is X-Robots-Tag equal to a meta robots tag?
For noindex and nofollow, yes — Google treats the header and the tag as the same directive through different channels. The header has the wider reach: it applies to any file type, including PDFs where a meta tag cannot exist.
Can I verify a single URL before the next crawl?
Yes — the free noindex check reports both channels on one URL, including the X-Robots-Tag header. This crawl rule is the site-wide sweep that catches the header on pages nobody thought to test.
Run this check now: Noindex checker
Free, no signup. Or run the full site audit on one URL.