Grok
xAI · reviewed 2026-11-05
About Grok
Generic Grok user agent seen in server logs; xAI has not documented which Grok product emits it.
Operates Grok; publishes several crawler user agents whose individual purposes are not fully documented.
Canonical directory entry: Grok-DeepSearch — xAI has not documented a distinct purpose for this alias.
Usually means: A generic Grok token fetched this URL. Treat it as Grok-family traffic; xAI has not documented which product or feature emits it.
Identity
User agent
Crawlers are matched by token, not the full browser string. Providers may add version or product text around it.
robots.txt token
Use this token in a User-agent line to allow or block the crawler.
Block this token
User-agent: Grok Disallow: /
A Disallow only stops crawlers that honor robots.txt. Classify the category before blocking; check the live file with the robots.txt checker.
Verify a hit
A matching user agent is not proof: any client can send the string. When the vendor publishes network data, compare the request against it before you trust the hit.
Match the user agent
Request UA contains Grok.
Check reverse DNS
The vendor publishes no reverse-DNS suffix.
Check the IP range
The vendor publishes no IP ranges.
A verified hit proves the network path; it does not prove what the crawler will do with the page.
Grok: common questions
What is Grok?
Grok is a crawler from xAI, filed under Other AI bots. Generic Grok user agent seen in server logs; xAI has not documented which Grok product emits it. Requests advertise it in the User-Agent header as Grok.
Does Grok train AI models?
The directory files it under Other AI bots: no training role is documented for it, and the vendor has not published enough to classify it further.
How do I block Grok in robots.txt?
Add a group with User-agent: Grok and Disallow: / to robots.txt, then re-test the live file. robots.txt matches the token, not the full User-Agent string, and blocking this token does not block the vendor's other crawlers.
How do I verify a Grok hit?
Only by user agent. xAI publishes neither reverse DNS nor IP ranges, so a match is not checked — that is not the same as forged, but it is not proof either.
Testing a block? Run the live file through the robots.txt tester or check the whole AI crawler access group.
See when Grok hits your site
Page scripts never see AI crawlers because they do not run JavaScript. Install the server probe and each hit lands in the Crawlers block with its path and status.