Validate your robots.txt
Check syntax errors, verify AI bot blocking (GPTBot, Claude, CCBot), validate IETF and Cloudflare signals, and test paths — before you deploy.
Try: · ·
How to check your robots.txt
Append /robots.txt to your domain — it's the first file every crawler requests.
Fetch or paste
Enter your domain to pull the live file, or paste a draft to test changes before you deploy.
AI bot audit
See which AI crawlers — GPTBot, ClaudeBot, CCBot, Google-Extended — are blocked, allowed, or not mentioned.
Test any path
After validation, check whether a specific URL path is allowed or denied for any user-agent.
Common robots.txt errors
Missing User-agent
Rules like Disallow: /admin/ must be preceded by a User-agent: line, otherwise bots will ignore them.
Relative sitemap URLs
Sitemap directives require absolute URLs. Sitemap: /sitemap.xml is invalid and will be ignored.
Blocking assets
Blocking /_next/ or /assets/ prevents Google from rendering your page correctly, hurting SEO.
Conflicting rules
Having both Allow: / and Disallow: / for the same agent can lead to unpredictable bot behavior.
Frequently asked questions
What does a robots.txt validator check?
Syntax errors, invalid directives, missing User-agent declarations, sitemap URL format, AI crawler rules, and Content-Usage / Content-Signal directives.
How do I know if my robots.txt is blocking AI bots?
After checking, review the AI crawler status table. It shows whether GPTBot, ClaudeBot, CCBot and others are blocked, allowed, or not explicitly mentioned.
How do I test if a specific URL is blocked?
Use the path tester at the bottom. Enter a path like /admin/ and pick a user-agent to see if access would be granted or denied.
Why is Google still crawling pages I blocked?
Robots.txt only suggests crawl behavior — it doesn't enforce it. Disallow prevents crawling but not indexing. Use meta robots noindex or password protection for true blocking.
Find out if AI is recommending you
One domain, five minutes, and you'll know exactly where you stand. *No card required.
Free report · No credit card · Results in ~5 minutes
Free tools
More free AEO tools
No sign-up, no catch. Built to help you get found, quoted and recommended by AI.