Checks included
- File availability and response status
- Risky global rules such as `Disallow: /`
- Presence of a sitemap directive
Results model
Robots.txt Checker result model
The report focuses on whether robots.txt is reachable, safe for crawling, and clear about sitemap discovery.
Recommended actions
- Remove blanket blocking rules from production sites.
- Keep important content and verification paths crawlable.
- Add a clean sitemap line pointing to the canonical host.