Crawler Governance

What it does

Crawler Governance helps you control which crawlers and AI bots can access your site. SEO Roger observes crawler activity and GA4 AI-referral signals, lets you decide a policy for each bot, then synthesizes the robots.txt and firewall rules needed to enforce that policy. It can even apply rules directly through the Cloudflare API and monitor for drift on a schedule.

How to use it

1. Open Tools > Crawler Governance.
2. On the Observe tab, review which crawlers and AI referrers have been hitting your site.
3. On the Policy tab, set each bot to Allow or Block based on your goals.
4. Generate your robots.txt and WAF rules from the policy.
5. Copy the output into your server, or connect Cloudflare to apply rules automatically.
6. Enable scheduled monitoring to be alerted if your live rules drift from the policy.

Reading the results

The Observe view shows each crawler with its recent request volume and whether it is a search engine, an AI trainer, or an AI answer engine. This tells you who is actually consuming your content. The Policy view then makes your intent explicit, and the generated files show exactly what will be enforced.

Blocking AI training bots while allowing search and answer engines is a common stance, but the right policy depends on whether you want AI visibility or content protection.

Credits

Crawler Governance is free, including observation, policy, file generation, and drift monitoring.