Crawler Governance

What it does

Crawler Governance helps you control which crawlers and AI bots can access your site. SEO Roger observes crawler activity and GA4 AI-referral signals, lets you decide a policy for each bot, then synthesizes the robots.txt and firewall rules needed to enforce that policy. It can even apply rules directly through the Cloudflare API and monitor for drift on a schedule.

How to use it

Reading the results

The Observe view shows each crawler with its recent request volume and whether it is a search engine, an AI trainer, or an AI answer engine. This tells you who is actually consuming your content. The Policy view then makes your intent explicit, and the generated files show exactly what will be enforced.

Blocking AI training bots while allowing search and answer engines is a common stance, but the right policy depends on whether you want AI visibility or content protection.

Credits

Crawler Governance is free, including observation, policy, file generation, and drift monitoring.