SEO crawler
SEO crawler: crawl the pages you audit, joined per URL
An SEO crawler answers a narrow question well: what does a search engine meet when it fetches this page? This one crawls the page set you nominate from the cloud, records the per-URL faults, and keeps each URL beside its Search Console clicks, Analytics sessions and AI crawler hits — so a crawl finding has a consequence, not just a severity.
Last reviewed 2026-11-18
The free check crawls one page set; scheduled crawls and page budgets are plan features — see pricing.
What an SEO crawler should answer
Four questions, four different pieces of evidence. The crawl keeps them separate so a clean status code cannot hide a page nobody links to, and a template fault cannot hide behind one healthy URL.
| Question | Where the answer comes from | Limit |
|---|---|---|
| Does the page return cleanly? | HTTP status, redirect chain, canonical target, robots directives, soft-404 sniffing | A 200 with an error page body is still a 200; the crawl flags it, it cannot fix it. |
| Can a crawler reach it? | Internal links, orphan candidates, sitemap membership, robots.txt access | A link proves a path exists, not that Google will follow or index it. |
| Does the template repeat the fault? | Titles, metas, H1, hreflang, structured data, image alt, mixed content | A template verdict needs enough crawled URLs of that template; small samples read na. |
| Did the fix change anything? | Run-over-run compare: added, fixed and regressed findings per URL | Needs a second crawl of the same page set; one run is a snapshot, not a trend. |
Per-URL findings
Crawl · sample rows
- /pricing200 · self-canonical · 1 H1Pass
- /blog/old-post301 → /blog/new-post · 1 hopWarn
- /collections/filternoindex in sitemap · canonical elsewhereFail
Sample data. The free check crawls a page set; scheduled crawls are plan features.
Crawl data joined per URL
A crawl on its own can only tell you what a page looks like. Joined to the sources you already own, the same row says whether the fault matters: a canonical loop on a page with 4,000 clicks is urgent; the same loop on a page with no impressions is a cleanup.
| Source | What it adds per URL | Window & limit |
|---|---|---|
| This crawl | Status, redirects, canonical, robots directives, titles, metas, headings, structured data, links, images | The page set of that run; a blocked URL is reported, never fetched. |
| Search Console | Clicks, impressions, CTR and average position for the same URL | One to three day lag; position is an average for the window. |
| GA4 or the page beacon | Sessions, engaged sessions and conversions | GA4 page detail is a seven-day top-20 snapshot; pages outside it read na. |
| AI crawler probe | Which AI agents fetched the URL, with status codes | Needs the server probe installed; a fetch is not a citation. |
| Tracked keywords | The queries the page is meant to answer, with volume and difficulty | Only the terms you added; no competitor keyword mining. |
How to run an SEO crawl
Step 1
Nominate the page set
Start from the site root or sitemap and add the URLs that carry revenue: product, pricing, docs, the pages Search Console already shows. The crawl stays inside the sites you verified — it does not wander into the open web.
Step 2
Crawl once, read by template
The first run tells you whether a fault is one page or a template. Sort findings by template and URL count; a duplicated title on 400 product pages is one fix, not 400.
Step 3
Join the crawl with the signals you already have
Each URL carries its Search Console clicks and position, GA4 sessions or conversions, AI crawler hits when the probe is installed, and its tracked keywords. The crawl explains the mechanism; the signals show the consequence.
Step 4
Fix, re-crawl, compare
Re-run the same page set after the deploy. The compare lists what was fixed, what regressed and what never changed, so a fix is verified rather than assumed.
SEO crawler vs desktop crawler
Both fetch pages; they are built for different jobs. A desktop crawler is the right tool for one exhaustive configuration pass. This crawler is built to be re-run, joined and compared — the shape an audit schedule needs.
| Dimension | This cloud crawler | Desktop crawler |
|---|---|---|
| Install | Nothing to install: the crawl runs from the cloud on the page set you name. | A desktop crawler runs on one machine, with its own RAM, CPU and crawl rate. |
| Scope | Audit-scoped: a page budget per run, with templates and sitemap discovery inside verified sites. | Can crawl a whole site exhaustively, including configurations you would not audit every week. |
| What it joins | Crawl findings sit beside Search Console, GA4, AI crawler hits, keywords and competitor changes for the same URL. | Crawl data leaves the tool as a CSV; joins to analytics and Search Console happen elsewhere. |
| Ongoing cost | Billed per monitored site, with the page budget stated before the run. | A licence per seat or per URL tier, plus the machine and the time to run it. |
What this crawler does not do
An SEO crawler is easy to over-read. These are the limits stated up front, not footnotes.
- It crawls the page set you nominate inside verified sites. It is not an open-web crawler and it does not follow links into other domains.
- It honors robots.txt: a Disallow rule keeps pages out of the crawl and the audit reports them as blocked instead of fetching them anyway.
- It does not ingest server logs or CDN logs. Crawl demand from real Googlebot traffic needs log files or the AI crawler probe; a crawl of your own is a different measurement.
- A crawl is not an index report. Whether Google indexed a URL is read from Search Console, not inferred from the crawl.
- Render parity is reported only where a render pass is available for the URL; JavaScript-only content can still read differently to a no-JS fetch.
Frequently asked questions
Definitions
What is an SEO crawler?
An SEO crawler fetches a site's pages the way a search engine would and records what each response contains: status, canonical, robots directives, titles, links and structured data. The output is a list of per-URL faults a crawler or a search engine would meet, not a ranking report.
How is this SEO crawler different from a desktop crawler?
It runs in the cloud, crawls the page set you nominate, and keeps the findings beside Search Console, Analytics and AI crawler data for the same URL. A desktop crawler is better when you need an exhaustive single-site configuration crawl; this one is built for a recurring audit you re-run and compare.
Is there a free SEO crawler here?
Yes: the free site check crawls a page set and returns findings with fixes, without a card. Scheduled crawls, larger page budgets and re-checks are plan features.
Scope & access
Which pages does it crawl?
The URLs you add to the audit list, plus sitemap discovery from the site root. The crawler stays inside sites you have verified and does not follow links into unrelated domains.
Does it crawl JavaScript-rendered content?
The base crawl reads the no-JS response. Where a render pass is available for a URL, the audit can compare the rendered DOM against the no-JS HTML; where it is not, render parity reads na instead of guessing.
Does it respect robots.txt?
Yes. A Disallow rule for VisdendaAuditBot or for all user agents keeps the page out of the crawl, and the run reports the URL as blocked. The crawler does not fetch a page it was told not to fetch.
Data & verification
What does the crawl join with?
Per URL: Search Console clicks, impressions and average position; GA4 sessions and conversions when it is bound; AI crawler hits when the probe is installed; tracked keywords; competitor site changes; and the findings themselves. Each source keeps its own window and reads na when missing.
How do I know a fix worked?
Re-crawl the same page set and use the run-over-run compare. It lists findings added, fixed and regressed per URL; a fix only counts once the re-check reaches the URL again.
Start free
Crawl one page set, then decide what to fix
Free includes 1 verified site and the page dossier; scheduled crawls and larger page budgets are plan features.