AI crawler analytics
AI crawler analytics: which agents fetch your site
A crawler log is the supply side of AI visibility: if GPTBot, ClaudeBot or PerplexityBot cannot fetch a page, no answer can cite it. This console reports agent hits, status codes, Markdown negotiation, machine-file reads, robots violations and 404s — per path, with the window stated.
Last reviewed 2026-11-13
Access checks are free; crawler analytics needs the probe installed — see pricing.
AI crawler analytics: four questions the report answers
Four questions, each with its own evidence. The report keeps them separate so a blocked path cannot hide behind a rising total.
| Question | Report block | Source | Limit |
|---|---|---|---|
| Which agents fetch | Hits by agent | Server probe or CDN logs | A user-agent string can be spoofed; an unknown agent is a lead, not proof. |
| What they read | Paths, Markdown negotiation, machine-file reads | Server probe | A read proves access, not that an answer used the page. |
| Where they are blocked | 403/404 and robots violations by path | Server probe plus access checks | A blocked path may be intentional; the report names the rule, not the intent. |
| Is the crawl changing | Hits per agent across windows | Server probe | A stable total can hide one agent leaving and another arriving. |
Crawler hits by agent
Probe · 7-day window
- GPTBot1,204 hits · 200 OKPass
- ClaudeBot318 hits · 7 blockedWarn
- PerplexityBot96 hits · 12 × 404Fail
Sample data. Probe required; access checks are free.
How to use AI crawler analytics
Step 1
Confirm the agent may fetch
Start with the access checks: robots.txt, content signals, bot identity and headers. A log full of 403s is a configuration problem, not an analytics problem.
Step 2
Find the paths with machine demand
Sort by hits and look for pages you do not consider citable: old URLs, parameter pages, staging hosts. Crawlers follow whatever they find; the report shows where they actually went.
Step 3
Fix the blocked and missing paths
Repair 404s, unblock the paths you want cited and redirect the ones you retired. Re-run the access checks after the deploy so the fix is verified, not assumed.
Step 4
Watch the frequency, not just the total
A stable total can hide one agent leaving and another arriving. Compare per-agent counts across windows before concluding that crawl demand is healthy.
What a crawl does not prove
Crawler analytics is easy to over-read. These are the limits the report states instead of smoothing over.
- A fetch is not a citation. It proves an agent read the page, not that an answer used it.
- The console reports crawler activity it observed; it does not see requests that never reached your site.
- User-agent strings can be spoofed. Treat an unknown agent as a lead, not as a verified vendor bot.
- No per-answer attribution. The report joins crawl activity to citations by page and window, not to a specific answer.
Frequently asked questions
Definitions
What is AI crawler analytics?
It is the measurement of which AI agents fetch your site, what they read, where they are blocked and which paths they request that no longer exist. It is the supply side of AI visibility: without a successful fetch, a citation is impossible.
Which AI crawlers can I monitor?
The public catalog includes GPTBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended and Bytespider, among others. The console reports the agents it actually observes.
How is this different from reading raw server logs?
The report is already grouped by agent, status and path, and it sits next to findings, citations and AI referrals for the same site. Raw logs are broader but need their own parsing, retention and joins before they answer an SEO question.
What a crawl proves
Does a crawler hit mean my page will be cited?
No. A fetch only proves the agent read the page. Whether an answer uses it depends on the engine, the query and the sources it selects; the citation sample is the separate measure.
Can I see how often GPTBot visits?
Yes: hits are grouped by agent and window, so you can compare frequency across periods. It is an observed count from your own site, not a schedule published by the vendor.
Blocking & access
How do I stop an AI crawler from fetching my site?
Use robots.txt rules for the agent token, declare content signals, and re-run the access check to confirm the rule is live. The crawler directory lists each token and the vendor documentation behind it.
Start with access
Check what AI crawlers can fetch today
One URL, no account. Crawler analytics needs the probe installed.