Tool · engine · active v1

robots.txt tester

One malformed line can hide your whole site from the crawler you want. The tester parses your robots.txt with the same checker the audit runs.

What it checks

Fetches your /robots.txt and checks syntax plus the Sitemap pointer.

Directive syntax, Sitemap: presence with an absolute http(s) URL.

How verdicts work

pass means parseable with at least one valid Sitemap pointer; malformed lines or a missing Sitemap fail; an unreachable file is na — absence itself is not a violation.

Reads the no-JS response only, never runs JS. Follows up to 5 redirects.

Official source: Search Central, robots.txt specifications · Rule: metrics-robots-txt v1

Example output

robots.txt parses, 1 Sitemap: entries

Related tools

llms.txt checker — Check your llms.txt for AI crawlers
JSON-LD validator — Paste JSON-LD, verify type and headline