Home / Checker reference / aeo-crawl-policy
Rule · engine · active v1
AI crawler policy check
active
engine
AI answers can only cite pages their crawlers may fetch and parse. One blocked crawler or one stale render and the citation goes to whoever ships cleaner machine-readable markup.
What it checks
Robots directives are resolved against the configured AI crawler set, /llms.txt is fetched and validated, and the JS-rendered DOM is diffed against the no-JS HTML for parity. Headings and freshness signals are read from the same fetch.
Pass conditions
- AI crawlers allowed, llms.txt valid, JS/no-JS parity ≥0.9, answers up front, fresh signals
- Crawler verdicts resolve per-UA from data, never from hardcoded UA lists.
- Render parity is measured, not assumed — one fetch per mode, same URL.
Typical fails
AI user-agents disallowed while the site claims to welcome AI citations; /llms.txt soft-404s (200 with HTML) or ships no valid links; JS-only answers with no no-JS fallback; pages whose freshest signal predates the last substantive update.
Fix checklist
- Allow the configured AI crawler set in robots; keep contradictions out of the directives.
- Serve a valid
llms.txt(reachable Markdown with an H1 and live links) — a clean 404 beats a soft 404. - Render answers server-side first, then enhance — parity ≥0.9 on every audited page.
Official source: llms.txt community convention and AI crawler documentation (demo placeholder link) · Status: engine, active v1, sole audit basis for AI crawler policy