Guide
Robots.txt basics for publishers and content sites
Understand which rules are safe, which ones are risky, and how to avoid blocking revenue-critical pages.
Robots.txt should guide crawlers, not blindly shut them out.
Guide crawlers instead of blocking everything
A production robots.txt file should help search engines discover content, not accidentally prevent indexing. Many publishers copy staging rules into production and create invisible traffic losses.
Keep verification resources open
Files such as `ads.txt`, key HTML pages, and sitemap references should remain publicly reachable. If you block them, monetization and indexing checks may fail even when the rest of the site looks fine.