Free tool group · no signup · AI crawler access
Robots.txt checker & llms.txt checker
Run a robots.txt checker and llms.txt checker in one pass. AI answers can only cite pages their crawlers may fetch and parse: the files and signals below decide whether your site is fetchable, trainable and citable.
Included in the free full scan — run this category alone when you only need these checks.
Want the AEO view? Run the AEO checker · Need a file? Generate an llms.txt · Want the log view? AI crawler analytics.
Checks site-wide files at the domain root — any page URL on the domain gives the same result.
One free scan of this category · anonymous preview · sign in to save the full report
What one scan checks here
Included in the scan above — open any check for the full criteria, evidence and fix notes.
- llms.txt checker — See whether AI crawlers can fetch, parse and cite your llms.txt.
- robots.txt tester — See whether your robots.txt parses cleanly and points crawlers at your sitemap.
- Content Signals checker — See what your site tells AI crawlers they may train on.
- Web-Bot-Auth checker — See whether bots can prove their identity with your published keys.
Frequently asked questions
What is an AI crawler access checker?
It reads the machine-facing files AI crawlers request — robots.txt, llms.txt, content signals and bot identity — and reports whether each one lets the crawler fetch, train on or cite your pages.
What is the difference between llms.txt and robots.txt?
robots.txt tells crawlers what they may not fetch; llms.txt tells AI models what they may cite. You need both: one guards access, the other feeds citations.
Do AI crawlers respect robots.txt?
The major ones declare user agents (GPTBot, ClaudeBot, CCBot) and honor disallow rules. The tester above parses your file the same way the audit does.
What each check verifies, and what a failure looks like, lives in the checker library.