AI Crawler Matrix
See which AI bots can crawl and cite you.
A clear matrix of which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended…) your site allows or blocks.
What you get
Everything AI Crawler Matrix delivers, out of the box.
All major AI crawlers
Allowed / blocked at a glance
robots.txt insight
Free to check
How it works
From input to insight in three steps.
Enter your URL or keyword
Point AI Crawler Matrix at any page, domain or search term, yours or a competitor's.
SEO Roger analyzes it
We run the analysis with Claude across the major AI engines in seconds.
Get a prioritized report
Read a clear, ranked breakdown of exactly what to do next.
Related tools
More from AI Visibility.
What is AI Crawler Matrix?
The AI Crawler Matrix reads your robots.txt and shows, in one grid, which AI crawlers your site allows and which it blocks: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest. It is free on every plan.
How to use AI Crawler Matrix
From signup to shipped fix, this is the whole loop.
- 1Create a free accountSign up in under a minute. No credit card is required, and your first site scan works without an account at all.
- 2Point AI Crawler Matrix at your siteEnter the URL or domain you want to analyze. You can run it on your own site or on a competitor's.
- 3Let it pull the live dataAI Crawler Matrix fetches the current state of the page or domain and runs its checks. Most runs finish in seconds.
- 4Read the ranked findingsResults come back ordered by impact rather than by count, so the issue costing you the most traffic is at the top, not buried under cosmetic warnings.
- 5Ship the fix, then re-runApply the recommended change, or let Roger AI draft it for you, then run the tool again to confirm the issue is resolved and track the improvement.
Why AI Crawler Matrix matters
A surprising number of sites block the AI crawlers they most want to be cited by, usually without knowing it. It happens through an inherited robots.txt, a security plugin's default rules, a CDN bot-protection setting or a well-meaning blanket block added during the training-data debates. An engine cannot cite content it was never allowed to fetch, so this is the first thing to check and the fastest thing to fix.
Search is no longer only Google. ChatGPT, Google AI Overviews, Perplexity, Gemini and Claude answer a growing share of queries by reading the same pages crawlers do, then citing a handful of sources. SEO Roger keeps both audiences in view: the classic SERP and the AI answer, from one workspace.
Who it's for
- Brands that want to be recommended by ChatGPT, Perplexity and AI Overviews, not just ranked in Google
- In-house SEOs who need visibility inside AI answer engines without stitching three tools together
- Agencies and consultants producing client-ready findings, fast
- Founders and marketers who want the fix, not just the metric
AI Crawler Matrix in depth
The crawlers do different jobs
Lumping them together produces bad decisions, because a single user agent often serves one purpose and not another. Some crawlers gather training data. Some fetch pages live to answer a question a user just asked. Some do both under different agent names. Notably, Google-Extended controls whether your content is used for Gemini and related AI products; it does not affect Googlebot or your ordinary search rankings at all. Confusing the two is a common and expensive mistake.
The real trade-off
There is a genuine decision here and it deserves an honest framing rather than a default. Allowing AI crawlers means your content can be used in ways you do not control, including to answer questions without sending you a visit. Blocking them means you are guaranteed not to be cited by that engine. For most businesses whose content is marketing rather than the product itself, visibility is worth more than the control. For publishers whose content is the product, the calculation is genuinely different and blocking can be rational.
Blocks are frequently accidental
Check the whole chain, not just the file you wrote. A wildcard disallow, an inherited rule from a previous agency, a security plugin adding bot rules, or a CDN's managed bot protection can each block AI crawlers without anything appearing in the robots.txt you maintain. This is why a matrix showing actual outcomes per agent is more useful than reading the file and assuming.
robots.txt is a request, not a wall
Compliant crawlers from established companies honour it. Anything determined to ignore it simply will, because robots.txt is a convention with no enforcement behind it. If you need actual enforcement rather than a polite request, that is a server or firewall job, which is what Crawler Governance handles. Treat robots.txt as the way you communicate intent to well-behaved bots.
AI Crawler Matrix questions, answered
Does this cost credits?
No. The AI Crawler Matrix is free on every plan. It reads your robots.txt, which costs us nothing to fetch.
Should I allow or block AI crawlers?
It depends on what your content is for. If your site exists to market a business, allowing them is usually right, because being cited builds awareness you cannot otherwise buy. If your content is itself the product, as with a publisher or a paid research service, blocking some crawlers is a defensible commercial decision. There is no universally correct answer, which is why the tool reports rather than prescribes.
Does blocking Google-Extended hurt my search rankings?
No. Google-Extended governs the use of your content in Gemini and related AI products. It is separate from Googlebot, and blocking it does not affect crawling, indexing or ranking in Google Search. This is the single most common misconception in this area.
Will allowing AI crawlers cost me traffic?
Possibly some, and it is fair to acknowledge that. An AI answer that fully satisfies a question may replace a visit you would otherwise have received. The counterweight is being present at the moment a recommendation is made, which for most businesses is worth more than the marginal informational visit. Weigh it against what your content is actually for.
How quickly does a robots.txt change take effect?
Crawlers recheck robots.txt periodically rather than on every request, so allow days rather than expecting immediate effect. Rerun the matrix after a change to confirm it reads as you intended.
What if I want to block bots that ignore robots.txt?
You need enforcement at the server or firewall layer, since robots.txt has no teeth. Crawler Governance generates the rules for that, including firewall configuration you can apply directly.
Ready to try AI Crawler Matrix?
Create a free account and run it in minutes, plus 55+ other SEO and AI-visibility tools in the same workspace.