SEO crawler

SEO crawler: crawl the pages you audit, joined per URL

An SEO crawler answers a narrow question well: what does a search engine meet when it fetches this page? This one crawls the page set you nominate from the cloud, records the per-URL faults, and keeps each URL beside its Search Console clicks, Analytics sessions and AI crawler hits — so a crawl finding has a consequence, not just a severity.

Last reviewed 2026-11-18

The free check crawls one page set; scheduled crawls and page budgets are plan features — see pricing.

What an SEO crawler should answer

Four questions, four different pieces of evidence. The crawl keeps them separate so a clean status code cannot hide a page nobody links to, and a template fault cannot hide behind one healthy URL.

SEO crawler questions, sources and limits
QuestionWhere the answer comes fromLimit
Does the page return cleanly?HTTP status, redirect chain, canonical target, robots directives, soft-404 sniffingA 200 with an error page body is still a 200; the crawl flags it, it cannot fix it.
Can a crawler reach it?Internal links, orphan candidates, sitemap membership, robots.txt accessA link proves a path exists, not that Google will follow or index it.
Does the template repeat the fault?Titles, metas, H1, hreflang, structured data, image alt, mixed contentA template verdict needs enough crawled URLs of that template; small samples read na.
Did the fix change anything?Run-over-run compare: added, fixed and regressed findings per URLNeeds a second crawl of the same page set; one run is a snapshot, not a trend.

Per-URL findings

Crawl · sample rows

  • /pricing200 · self-canonical · 1 H1
    Pass
  • /blog/old-post301 → /blog/new-post · 1 hop
    Warn
  • /collections/filternoindex in sitemap · canonical elsewhere
    Fail

Sample data. The free check crawls a page set; scheduled crawls are plan features.

Crawl data joined per URL

A crawl on its own can only tell you what a page looks like. Joined to the sources you already own, the same row says whether the fault matters: a canonical loop on a page with 4,000 clicks is urgent; the same loop on a page with no impressions is a cleanup.

Per-URL crawl joins, windows and limits
SourceWhat it adds per URLWindow & limit
This crawlStatus, redirects, canonical, robots directives, titles, metas, headings, structured data, links, imagesThe page set of that run; a blocked URL is reported, never fetched.
Search ConsoleClicks, impressions, CTR and average position for the same URLOne to three day lag; position is an average for the window.
GA4 or the page beaconSessions, engaged sessions and conversionsGA4 page detail is a seven-day top-20 snapshot; pages outside it read na.
AI crawler probeWhich AI agents fetched the URL, with status codesNeeds the server probe installed; a fetch is not a citation.
Tracked keywordsThe queries the page is meant to answer, with volume and difficultyOnly the terms you added; no competitor keyword mining.

How to run an SEO crawl

  1. Step 1

    Nominate the page set

    Start from the site root or sitemap and add the URLs that carry revenue: product, pricing, docs, the pages Search Console already shows. The crawl stays inside the sites you verified — it does not wander into the open web.

  2. Step 2

    Crawl once, read by template

    The first run tells you whether a fault is one page or a template. Sort findings by template and URL count; a duplicated title on 400 product pages is one fix, not 400.

  3. Step 3

    Join the crawl with the signals you already have

    Each URL carries its Search Console clicks and position, GA4 sessions or conversions, AI crawler hits when the probe is installed, and its tracked keywords. The crawl explains the mechanism; the signals show the consequence.

  4. Step 4

    Fix, re-crawl, compare

    Re-run the same page set after the deploy. The compare lists what was fixed, what regressed and what never changed, so a fix is verified rather than assumed.

SEO crawler vs desktop crawler

Both fetch pages; they are built for different jobs. A desktop crawler is the right tool for one exhaustive configuration pass. This crawler is built to be re-run, joined and compared — the shape an audit schedule needs.

SEO crawler compared with a desktop crawler by install, scope, joins and cost
DimensionThis cloud crawlerDesktop crawler
InstallNothing to install: the crawl runs from the cloud on the page set you name.A desktop crawler runs on one machine, with its own RAM, CPU and crawl rate.
ScopeAudit-scoped: a page budget per run, with templates and sitemap discovery inside verified sites.Can crawl a whole site exhaustively, including configurations you would not audit every week.
What it joinsCrawl findings sit beside Search Console, GA4, AI crawler hits, keywords and competitor changes for the same URL.Crawl data leaves the tool as a CSV; joins to analytics and Search Console happen elsewhere.
Ongoing costBilled per monitored site, with the page budget stated before the run.A licence per seat or per URL tier, plus the machine and the time to run it.

What this crawler does not do

An SEO crawler is easy to over-read. These are the limits stated up front, not footnotes.

  • It crawls the page set you nominate inside verified sites. It is not an open-web crawler and it does not follow links into other domains.
  • It honors robots.txt: a Disallow rule keeps pages out of the crawl and the audit reports them as blocked instead of fetching them anyway.
  • It does not ingest server logs or CDN logs. Crawl demand from real Googlebot traffic needs log files or the AI crawler probe; a crawl of your own is a different measurement.
  • A crawl is not an index report. Whether Google indexed a URL is read from Search Console, not inferred from the crawl.
  • Render parity is reported only where a render pass is available for the URL; JavaScript-only content can still read differently to a no-JS fetch.

Frequently asked questions

Definitions

What is an SEO crawler?

An SEO crawler fetches a site's pages the way a search engine would and records what each response contains: status, canonical, robots directives, titles, links and structured data. The output is a list of per-URL faults a crawler or a search engine would meet, not a ranking report.

How is this SEO crawler different from a desktop crawler?

It runs in the cloud, crawls the page set you nominate, and keeps the findings beside Search Console, Analytics and AI crawler data for the same URL. A desktop crawler is better when you need an exhaustive single-site configuration crawl; this one is built for a recurring audit you re-run and compare.

Is there a free SEO crawler here?

Yes: the free site check crawls a page set and returns findings with fixes, without a card. Scheduled crawls, larger page budgets and re-checks are plan features.

Scope & access

Which pages does it crawl?

The URLs you add to the audit list, plus sitemap discovery from the site root. The crawler stays inside sites you have verified and does not follow links into unrelated domains.

Does it crawl JavaScript-rendered content?

The base crawl reads the no-JS response. Where a render pass is available for a URL, the audit can compare the rendered DOM against the no-JS HTML; where it is not, render parity reads na instead of guessing.

Does it respect robots.txt?

Yes. A Disallow rule for VisdendaAuditBot or for all user agents keeps the page out of the crawl, and the run reports the URL as blocked. The crawler does not fetch a page it was told not to fetch.

Data & verification

What does the crawl join with?

Per URL: Search Console clicks, impressions and average position; GA4 sessions and conversions when it is bound; AI crawler hits when the probe is installed; tracked keywords; competitor site changes; and the findings themselves. Each source keeps its own window and reads na when missing.

How do I know a fix worked?

Re-crawl the same page set and use the run-over-run compare. It lists findings added, fixed and regressed per URL; a fix only counts once the re-check reaches the URL again.

Start free

Crawl one page set, then decide what to fix

Free includes 1 verified site and the page dossier; scheduled crawls and larger page budgets are plan features.