Crawler directory
AI crawler & search bot directory
63 user agents that fetch pages for AI answers, AI search, search engines, training and ads — with the robots.txt token, the documented user agent, reverse-DNS suffixes where the vendor publishes them, and a link to the vendor's own documentation.
Directory 2026-11 · last human review 2026-11-13 · every entry is in the HTML, no JavaScript required
AI crawler list: 63 user agents
Browse 63 crawlers by category or provider, or search a user agent. Every entry is in the HTML — no JavaScript required.
63 of 63 crawlers
AI answers
User-triggered / 15 crawlers
Fetched when a person asks an assistant a question that needs live context. One page per question, not a crawl.
Qwen-User
Fetches a page when a Qwen user asks a question needing web context.
Amzn-User
Fetches a page for an Amazon AI assistant answering a user question.
Claude-User
Fetches a page when a Claude user asks for up-to-date web information.
DuckAssistBot
Fetches pages to draft Duck.ai's AI-assisted answers.
Google-Agent
Fetches a page when a Google agent product runs a user task that needs it.
Google-NotebookLM
Fetches sources a user adds to a NotebookLM notebook.
Google-Read-Aloud
Fetches a page to read it aloud to a user.
GoogleAgent
Alias user agent for Google agent fetches; match both tokens to avoid split rows.
Meta-ExternalFetcher
Fetches a page when a Meta AI user asks a question needing web context.
Copilot
Fetches a page when a Microsoft Copilot user asks a question needing web context.
MistralAI-User
Fetches a page when a Le Chat user asks a question needing web context.
Kimi-User
Fetches a page when a Kimi user asks a question needing web context.
ChatGPT-User
Fetches a page when a ChatGPT user asks a question that needs live web context.
Perplexity-User
Fetches a page when a Perplexity user asks a question that needs the cited page.
Grok-DeepSearch
Fetches pages during a Grok user's deep search run.
AI search
AI discovery / 10 crawlers
Indexing crawlers behind AI search results and citations. Their index decides whether your page can be quoted.
Amzn-SearchBot
Indexes pages for Amazon's AI search surfaces.
Claude-SearchBot
Indexes pages for Claude's web search.
Applebot
Powers Apple's search surfaces (Siri, Spotlight) and Apple Intelligence products.
meta-webindexer
Indexes pages for Meta AI's web index.
MistralAI-Index
Indexes pages for Mistral's search products.
Kimi-SearchBot
Indexes pages for Kimi's web search.
OAI-SearchBot
Indexes pages for ChatGPT search results.
PerplexityBot
Indexes pages for the Perplexity answer engine.
xAI-SearchBot
Indexes pages to appear in Grok's search results.
YouBot
Indexes pages for You.com's answer engine.
Search engines
Classic index / 7 crawlers
Traditional indexers. They feed classic search results and are kept separate from the AI-arrival numbers.
Baiduspider
The main Baidu Search indexing crawler.
TikTokSpider
Indexes web content for TikTok's in-app search.
Google-InspectionTool
Google's debugging crawler behind Search Console live tests and inspections.
Googlebot
The main Google Search indexing crawler.
BingBot
The main Bing indexing crawler.
msnbot
Legacy Microsoft crawler that still appears in logs and feeds MSN/Bing infrastructure.
YandexBot
The main Yandex Search indexing crawler.
Training
Model data / 22 crawlers
Collect content for model training. Blocking them changes what future models learn from your site, not what today's answers cite.
QwenBot
Collects public pages to train Qwen models.
TongyiBot
Collects public pages for Alibaba's Tongyi models.
AI2Bot
Allen AI's research crawler collecting open corpora.
Amazonbot
Collects public pages for Amazon's AI services.
ClaudeBot
Collects public pages that may be used to train Anthropic models.
Applebot-Extended
robots.txt opt-out token for Apple's AI training; has no user agent of its own.
ERNIEBot
Collects public pages for Baidu's ERNIE models.
YiyanBot
Collects public pages for Baidu's Yiyan (ERNIE Bot) assistant.
Bytespider
ByteDance's training crawler; the noisiest AI crawler on many logs.
Doubaobot
Collects public pages for ByteDance's Doubao assistant.
cohere-ai
robots.txt opt-out token for Cohere; has no user agent of its own.
cohere-training-data-crawler
Collects public pages for Cohere training datasets.
CCBot
Fetches pages for Common Crawl's open quarterly web corpus.
DeepSeekBot
Collects public pages to train DeepSeek models (UA-only, no published IP ranges).
Google-CloudVertexBot
Fetches public pages for Vertex AI customers building models.
Google-Extended
robots.txt opt-out token for Gemini app training; has no user agent of its own (fetches come as Googlebot).
GoogleOther
Generic Google crawler used by unspecified product teams.
FacebookBot
Collects public pages to train Meta's AI models.
Meta-ExternalAgent
Collects public pages to train Meta's AI models.
KimiBot
Collects public pages to train Kimi models.
GPTBot
Collects public pages that may be used to train OpenAI models.
ChatGLM-Spider
Collects public pages to train GLM/ChatGLM models.
Ads
Ad review / 2 crawlers
Crawlers used for ad quality, landing-page review and policy checks.
Other AI bots
Unclassified / 7 crawlers
Crawlers whose vendor has not published enough to place them in a tighter category. Not a judgement about legitimacy.
AliyunBot
Alibaba Cloud crawler observed in the wild; purpose not officially documented.
facebookexternalhit
Fetches link previews when users share URLs on Meta apps; not AI training, listed for classification completeness.
Grok
Generic Grok user agent seen in server logs; xAI has not documented which Grok product emits it.
GrokBot
Grok-branded bot token observed in the wild; xAI has not published whether it fetches for answers, search or training.
xAI-Bot
Short xAI bot token observed in logs; it is not covered by xAI's public crawler documentation.
xAI-Grok
Hyphenated xAI/Grok token seen in logs; no official document ties it to a specific Grok surface.
xAI-Web-Crawler
Descriptive xAI crawler token seen in the wild; xAI has not documented its crawl purpose.
Crawler check: is this bot really Google?
A user agent is a claim, not proof — any client can send Googlebot. Google publishes its crawler IP ranges and reverse-DNS suffixes, and the same three-step check works for the vendors in this directory: match the token, resolve the hostname, then compare the IP against the vendor’s published ranges.
Run the free AI crawler access check to see which tokens your robots.txt and llms.txt allow. For a live hit on your own server, the server probe records the user agent, path and status where page scripts cannot see it.
xAI note: Grok, GrokBot, xAI-Bot, xAI-Grok and xAI-Web-Crawler are listed separately because each token appears in the wild. xAI has not documented a distinct purpose for each, so their pages point their canonical at the documented Grok entry; use the token you actually see in your logs.
Crawler questions, answered
What each category means, why some vendors have no reverse DNS, and how to tell a real hit from a spoofed user agent.
What is an AI crawler directory?
It is the public list of user agents that fetch pages for AI answers, AI search, model training and ads. For each one you get the robots.txt token, the documented user agent, reverse-DNS suffixes when the vendor publishes them, and a link to the vendor's own documentation.
How do I allow or block a crawler?
Use the robots.txt token: a Disallow line for that token blocks it, and an explicit Allow line keeps it fetchable. Classify before you block — training crawlers and answer-time fetchers have different effects, and blocking an AI search crawler also removes your page from that index.
Why does a crawler show 'not published' for reverse DNS?
Several vendors publish IP ranges but not reverse-DNS suffixes. Without a published suffix we cannot verify a hit by hostname, so the console reports it as not checked. That is not evidence of forgery: it only means the vendor gave us nothing to check against.
How fresh is this list?
Every entry carries the date a human last checked it against the vendor's documentation. Entries older than 90 days get a stale badge until they are re-reviewed. The directory version and the newest review date are also in /crawlers.json.
Why is Googlebot not counted as AI traffic?
Googlebot is the classic Search indexer; it is kept in its own search-engine category. AI answer fetches and AI search crawlers are counted separately, so a busy classic crawl never inflates the AI-arrival number.
How do I see which of these crawlers hit my site?
Install the server probe. Page scripts never see AI crawlers because they do not run JavaScript; a server-side probe records the user agent, path and response status. The npm package wraps a fetch handler (Workers, Bun, Deno, edge runtimes), a Node server or Next.js middleware in one line.
See which of these hit your site
Page scripts never see AI crawlers because they do not run JavaScript. Install the server probe and each hit lands in the Crawlers block with its path and status.