Torn on this. Part of me wants to block GPTBot and the others so they can't train on my content for free. The other part worries that if I block them I vanish from AI answers entirely and hand my visibility to competitors who don't block. What's the actual trade-off here? Curious how others are thinking about it.
This is exactly the decision Crawler Governance was built to make deliberate instead of accidental. The key insight most people miss: training crawlers and answer crawlers aren't always the same bot, and you can treat them differently.
The trade-off in plain terms:
Use the observe step to see who's actually crawling you and what referral traffic AI is already sending, then set per-crawler policy, then let it synthesize the robots.txt. Decide with data, not vibes.
The 'invisibility is the bigger cost' point is where I landed too. For a content business trying to get discovered, blocking the answer bots is shooting yourself in the foot. I allow retrieval, and I'm still on the fence about training bots.
Didn't even realize the answer bots and training bots could be split. That reframes the whole decision. Allowing the ones that get me cited, thinking harder about the rest. Thanks Devin.
