Guides

Googlebot index: what it fetches, renders and indexes

“Googlebot index” is usually asked as one question and answered as three. A page can be fetched and not rendered, rendered and not indexed, or indexed and never shown. Each stage has its own evidence, and only one of them is a content problem.

By the Visdenda research team · Reviewed by the Visdenda editorial team

Published 2026-11-18 · Updated 2026-11-18 · 8 min read

The three stages

  1. Fetch. Googlebot requests the URL from the crawl queue. The response — status, headers, robots rules — decides whether there is anything to process.
  2. Render. A page that needs JavaScript enters the render queue. The rendered DOM is what Google evaluates, so a no-JS-only answer can differ from what a user sees.
  3. Index. The processed page is stored and can be selected for results. Indexing is a decision, not a receipt: a fetched, rendered page can still be excluded.

Most “indexing problems” are one stage mistaken for another. A 404 is a fetch problem; an empty rendered shell is a render problem; “crawled — currently not indexed” is an index decision made after both.

Where to read each stage

  • Fetch volume and status. The Crawl stats report shows requests by response code and by file type, plus host status. It answers “is Googlebot crawling at all”, not “will this page index”.
  • One URL. URL Inspection shows the last crawl, the crawled page, the rendered HTML and the index verdict for that URL. This is the only page-level evidence Google publishes.
  • The set. The Pages report groups verdicts — indexed, crawled-not-indexed, discovered-not-indexed, excluded by directive — so a single page can be read against the site.
  • Your own headers. noindex, canonical and robots rules are on your side of the fence. Check them directly instead of inferring from a verdict.

The index status check reads the page-side signals — robots directives, canonical, status — in one run, so the directive half of the question is answered before you read Google's verdict.

Crawled, not indexed

This verdict means Google fetched the page and chose not to store it. The page-level causes are checkable: a noindex that the crawl also sees, a canonical pointing at another URL, a body that duplicates a stronger page, or a page that adds nothing unique to the set. The reasons Google does not publish — authority, freshness, demand — are not checkable, and guessing at them is how a fix turns into a rewrite nobody can verify.

Start with the directives, then compare the page to the URL it competes with. The duplicate content check and the thin content check explain what the crawl can prove before you change copy.

Blocked, noindexed, canonicalised

These three are often treated as synonyms and they are not. A robots block stops the fetch; the URL can still appear, without a snippet, because Google cannot read the page. noindex requires a fetch: the crawler must read the directive to honour it. A canonical is a hint: Google reads it and decides.

  • To remove a page: let Googlebot fetch it and serve noindex, or return 404/410.
  • To keep a page out of the index from day one: noindex, not a robots block.
  • To consolidate duplicates: canonical plus internal links that agree with it.

The noindex explainer covers the blocked-versus-noindex distinction in detail, including the case where a blocked page stays in the index for months.

Four myths that cost weeks

  • “Sitemap = indexed.” A sitemap is a discovery hint. Listing a URL does not fetch, render or index it.
  • “Request indexing fixes it.” It queues a fetch. If the page is noindexed or canonicalised elsewhere, the request succeeds and the verdict stays the same.
  • “Crawl budget explains it.” Crawl budget matters for large, frequently changing sites. For a hundred pages, the constraint is usually directives, duplication or demand — not crawl allocation.
  • “It will index eventually.” Sometimes true. But “eventually” without a date is not a diagnosis, and a page that has sat in discovered-not-indexed for weeks has a discoverability or quality problem worth naming.

Triage a URL in order

  1. Check the status code and redirect chain: a 3xx or 4xx ends the analysis.
  2. Read robots and meta robots: a block or noindex ends it too, with a known fix.
  3. Read the canonical: if it points elsewhere, decide which URL should win.
  4. Compare the body to the likely competitor URL: duplicate or thin is a content fix.
  5. Read URL Inspection for the rendered page; if the shell is empty, the render failed.
  6. Only then treat it as a demand or authority question — and say so honestly.

After the fix, re-check the same URL and keep the evidence. The verification method applies: a complete window, a baseline and a before/after reading — not a screenshot of a green badge.

Frequently asked questions

What is the Googlebot index?

There is no separate “Googlebot index”. Googlebot fetches and renders pages; the index is the store Google builds from what it processed. When people say Googlebot index they usually mean index status for the URLs Googlebot fetched — which is read in Search Console, not measured by watching the crawler.

How long does Googlebot take to index a new page?

There is no promised window. A linked page on a site with crawl demand can be fetched within hours; an orphan on a low-authority site can wait weeks. Request indexing moves a URL into the queue, it does not guarantee a fetch or an index entry.

Does blocking Googlebot remove a page from the index?

No. A robots.txt block stops the fetch, not the existing entry: Google can still show the URL with a snippet-less result because it cannot read the page. To remove a page from the index, return noindex (or a 404/410) while the crawler can still fetch it.

Why is a page crawled but not indexed?

The usual causes are a noindex directive, a canonical pointing elsewhere, a duplicate or thin body, or a quality decision Google does not explain. Read the page-level status in URL Inspection first; that separates a directive problem from a quality verdict.

Read the evidence

Check one URL's index signals without guessing

Free includes 1 verified site and the page dossier; index status is read from Search Console.