New: Crawler Governance — observe, decide, and enforce who crawls you

Announcing Crawler Governance, now available in the platform.

AI crawlers are hitting your servers whether you've thought about them or not. This feature gives you a full workflow instead of a guess:

1. Observe — real crawler access data plus GA4 AI-referral traffic so you can see who's crawling and what it's sending back.
2. Decide — set a policy per crawler: allow, block, or allow-with-limits.
3. Synthesize — we generate the exact robots.txt to match your policy.
4. Enforce — generate WAF rules to paste in, or apply them directly via the Cloudflare API if you connect it.
5. Monitor — scheduled drift detection alerts you when your live robots.txt or rules stop matching your intended policy.

This pairs naturally with the AI Visibility suite: use AI Visibility to decide who you want, then Crawler Governance to make it real and keep it that way.

This one is locked for replies while we route feedback through the product channel, but the thread will stay up as a reference.