Guides
Log file analysis for SEO: what logs prove that a crawl cannot
A crawl shows what a crawler could fetch. A log shows what Googlebot actually did — which URLs, how often, and what your server answered. The two disagree more often than people expect, and the log is the one that happened.
By the Visdenda research team · Reviewed by the Visdenda editorial team
Published 2026-11-18 · Updated 2026-11-18 · 8 min read
What logs answer that a crawl cannot
A crawl is a simulation: your crawler fetches the pages you nominated. A log is the record of what Googlebot requested and what your server returned. Four questions differ:
- Which URLs actually get crawled. Often not the ones in the sitemap: parameter pages, retired URLs, staging hosts.
- What status Googlebot received. A 5xx burst during a deploy is invisible in a crawl run an hour later.
- How often. Crawl frequency per directory is a demand signal you cannot infer from an audit.
- What is wasting crawl. Facets, calendars, infinite filters: the log shows the paths you did not intend to publish.
Verify the bot before you analyse anything
The user-agent string is a claim. Filtering a log on “Googlebot” without DNS verification mixes real Googlebot with scrapers and monitoring tools, and the conclusions follow the noise. Resolve each IP in reverse, keep the requests whose hostname ends in googlebot.com or google.com, then forward-resolve that hostname and confirm it returns the original IP. Cache the result per IP for the analysis window.
Do the same for AI agents if you analyse them: GPTBot, ClaudeBot and PerplexityBot each publish verification guidance, and a spoofed AI crawler is the easiest way to invent a trend. The AI crawler analytics path records agent hits at the server for exactly this reason.
Read the log in four passes
- Status first. Group verified Googlebot requests by response code. A 5xx share above noise is a server problem; a 404 share is a discovery problem; 200s tell you what is being fetched.
- Directory second. Requests per section show where crawl attention goes. If 60% is on filters and pagination, the catalogue is spending its budget on URLs nobody searches.
- Frequency third. Compare per-URL request counts across two windows. A page fetched daily that never changes is wasting crawl; a page that changes weekly and is fetched monthly is under-crawled.
- Join fourth. Attach clicks and impressions from Search Console to the same paths. Crawl demand without demand is a config question; crawl demand with demand is where the fixes pay.
Join logs with Search Console honestly
The log covers every request; Search Console covers what Google chose to report. The windows do not align, the URL normalisation differs, and clicks lag. Join on a normalised path, state the window, and read the two numbers as different measurements: fetch count is crawl attention, impressions are eligibility, clicks are demand.
A page with 300 fetches and zero impressions is usually a directive or duplication problem — the crawler comes back, the index does not. The fetch/render/index breakdown is the right frame for that case.
Four common misreads
- “Crawl budget is the problem.” For most sites under a few thousand URLs, it is not. Directives and URL sprawl explain more than allocation.
- “More fetches means more ranking.” Fetch frequency follows demand as much as it creates it. A page crawled more is not automatically indexed more.
- “The log shows all bots.” Logs show requests that reached your origin. CDN and bot-management layers can drop traffic before it arrives, so the log is a floor, not a census.
- “One week is a trend.” Deploys, launches and holidays move crawl patterns. Compare complete windows and keep the deploy dates beside the chart.
Fix and re-measure
The fixes are ordinary: block or noindex the parameter space, fix the 5xx source, retire the URLs you no longer want, and link the pages you do want crawled. Then re-measure the same log window: the share of 200s on money directories should rise, the parameter share should fall, and the fetches per changed URL should reflect how often the page actually changes.
Keep the crawl beside the log. The crawler tells you what the page looks like now; the log tells you what Googlebot met. A fix that changes one and not the other is worth a second look before it is called done.
Frequently asked questions
Is log file analysis still useful in 2026?
Yes, for sites where crawl demand is a real constraint: large catalogues, faceted stores, sites with frequent deploys. For a small site, Search Console's Crawl stats and URL Inspection answer most of the same questions with less setup. The log is the raw record; the report is a summary.
How do I know a request is really Googlebot?
Resolve the IP with reverse DNS and confirm the hostname ends in googlebot.com or google.com, then forward-resolve that hostname and check it matches the original IP. User-agent strings are trivial to spoof; DNS is not.
Can Search Console replace log files?
Not entirely. Crawl stats aggregates request volume by response and file type over time, but it does not give you per-URL raw requests, and it includes sampling. Logs give exact paths and statuses per request, without Google's aggregation.
Does a Googlebot fetch mean the page will rank?
No. A fetch proves the URL entered the crawl queue and the server answered. Indexing is a separate decision, and ranking is a third. Logs measure the first step only.