Check explainer · v1

X-Robots-Tag: the noindex you cannot see in the HTML

Automated

Last reviewed

The header says noindex and the HTML says nothing. Anyone checking the page source sees a clean page — and the site quietly drops out of the index.

How this check works

Every fetched page’s X-Robots-Tag response header is read for noindex or none. A match fails the page with the header value quoted: the directive is equivalent to a meta robots noindex, but it lives at the HTTP layer — invisible to page-source views and to tools that only read HTML.

What does a failure look like?

A CDN or edge rule that stamps X-Robots-Tag: noindex on a whole environment — staging config promoted to production; an auth middleware that marks every response; a bot-protection layer that answers with noindex on suspicious traffic — links and sitemaps intact, indexing silently off.

How to fix it, step by step

  1. Find the layer that sets the header: edge worker, CDN rule, reverse proxy or application middleware.
  2. Scope the directive to the environments that need it — staging yes, production no — and redeploy.
  3. Re-check a sample URL with the single-page noindex check and confirm the header is gone.

What does a pass or fail look like?

Good

No page carries noindex or none in its X-Robots-Tag response header.

Fail

Every response ships X-Robots-Tag: noindex, nofollow from an edge rule — the HTML is clean, and the whole site is excluded.

Frequently asked questions

Why can page-source tools miss this entirely?

Because X-Robots-Tag is a response header, not page markup. Tools that fetch and parse HTML never see it, so a site can look perfectly indexable in every page-source view while the header excludes everything.

Is X-Robots-Tag equal to a meta robots tag?

For noindex and nofollow, yes — Google treats the header and the tag as the same directive through different channels. The header has the wider reach: it applies to any file type, including PDFs where a meta tag cannot exist.

Can I verify a single URL before the next crawl?

Yes — the free noindex check reports both channels on one URL, including the X-Robots-Tag header. This crawl rule is the site-wide sweep that catches the header on pages nobody thought to test.

Run this check now: Noindex checker

Free, no signup. Or run the full site audit on one URL.

Visdenda check explainers · one page per published ruleX-Robots-Tag · v1