Robots.txt Validation: Every Directive, Every Common Mistake
Validate robots.txt syntax, understand Allow/Disallow/Sitemap, and learn why robot.txt (singular) is ignored — plus how AI crawlers read your rules.
Staging shipped with Disallow: / still in place, or the file is named robot.txt and nobody notices until search traffic flatlines. A validator fetches /robots.txt and shows which paths each user-agent may crawl, plus syntax errors that make crawlers ignore the file.
robots.txt vs robot.txt
The only valid name is robots.txt — plural, lowercase — at the site root (https://example.com/robots.txt). robot.txt is a frequent typo; major crawlers (including GPTBot, ClaudeBot, and PerplexityBot when configured) will not treat it as your policy file.
Validate in the browser
- Open the robots.txt analyzer.
- Enter the origin (
https://example.com, no path). - Read Allow/Disallow blocks per
User-agent,Sitemaplines, and parser errors. - Confirm production did not inherit a blanket
Disallow: /from staging.
Directives
| Directive | Role |
|---|---|
| User-agent | Which crawler the next rules apply to |
| Disallow | Paths the agent should not fetch |
| Allow | Exceptions inside a broader Disallow |
| Sitemap | Pointer to your XML sitemap |
| Crawl-delay | Hint — many crawlers ignore it |
robots.txt is not access control
Well-behaved bots honor the file. Attackers and misconfigured scrapers do not. Sensitive paths need authentication and authorization, not a Disallow line. When you have fixed auth, AI VAPT can exercise the app — robots rules alone prove nothing about exploitability.