Using the AI Crawler Matrix to control bot access
What it does
The AI Crawler Matrix gives you a clear grid of the major AI and search crawlers and whether each one can currently access your site. Answer engines can only cite content their crawlers are allowed to read, so a bot you accidentally block is content you have made invisible to that engine. The matrix turns a tangle of user agents and robots rules into a simple allow-or-block view.
How to use it
- Open Tools > AI Crawler Matrix.
- Enter your Domain.
- Click Check access.
- Review the grid: each row is a crawler and its Allowed or Blocked status, grouped into search engines, AI answer engines, and AI training bots.
- To change any status, jump to Crawler Governance to set policy and generate the robots.txt and firewall rules that enforce it.
Reading the results
Each crawler is labeled by type so you can make an informed choice. Search engine and AI answer engine bots are the ones that drive visibility and citations; blocking them usually hurts. AI training bots consume content to train models without necessarily sending traffic back, so some brands choose to block those while keeping answer engines open.
Look for surprises, such as an answer-engine crawler blocked by an overly broad robots rule. Those are the highest-priority fixes, since unblocking them can restore your eligibility to be cited.
Credits
Checking access in the AI Crawler Matrix is free. Enforcing changes through Crawler Governance is also free.
Was this helpful?