AI Crawler Access: GPTBot, ClaudeBot, PerplexityBot and Your robots.txt

Every AI assistant that can cite your content first has to crawl it. If your robots.txt quietly blocks the wrong bot, you can vanish from AI answers without ever seeing an error. This guide maps the major AI crawlers and shows how to make deliberate access decisions.

The crawlers you need to know

Different bots serve different purposes, and blocking one is not the same as blocking another:

• GPTBot (OpenAI): crawls content used to train and improve models.
• OAI-SearchBot (OpenAI): powers ChatGPT search results and citations.
• ClaudeBot (Anthropic): crawls content for Claude's training and retrieval.
• PerplexityBot (Perplexity): indexes pages for Perplexity's answer engine.
• Google-Extended (Google): a control token for Gemini and AI training, separate from Googlebot.
• Bingbot / Microsoft crawlers: feed Copilot and Bing's AI answers.

The critical nuance: Google-Extended is not Googlebot. Blocking Google-Extended does not remove you from Google Search, but it does opt you out of certain AI uses. Blocking Googlebot removes you from Search entirely. Confusing the two is a common and costly mistake.

Training crawlers vs search crawlers

A useful distinction:

1. Training crawlers (GPTBot, ClaudeBot, Google-Extended) gather content to improve models. Blocking these protects content from training but usually does not affect whether you're cited in live answers.
2. Search/answer crawlers (OAI-SearchBot, PerplexityBot, Bing) fetch pages to build real-time answers with citations. Blocking these directly removes you from those AI answers.

If your goal is AI-search visibility, be careful: you can allow the search bots while making a separate decision about training bots.

How to control access in robots.txt

robots.txt uses user-agent blocks. To allow ChatGPT search but block training, for example:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Allow: /

A few rules to remember:

• Directives are per user-agent; there is no wildcard that covers "all AI bots."
• Disallow: / blocks everything for that agent; an empty Disallow: allows everything.
• robots.txt is a request, not enforcement. Well-behaved bots honor it; if you need hard blocking, use server-level or WAF rules.

Decide with intent, not by accident

Before you edit anything, ask:

• Do we want to appear in AI answers? Then keep the search/answer bots allowed.
• Do we object to training use? Then consider blocking the training bots specifically.
• Are we accidentally blocking a bot via a broad rule or a security plugin? This is more common than teams expect.

See every bot's status at a glance

Reading robots.txt line by line across a dozen user agents is error-prone. SEO Roger's AI Crawler Matrix checks your live robots.txt and server responses for every major AI crawler and shows a clear allowed/blocked grid, so you can spot an accidental block in seconds. Pair it with the Full Site Audit to catch server-level blocks that robots.txt inspection alone would miss.

Monitor for drift

Access isn't set-and-forget. Security plugins, CDN rules, and well-meaning developers change things. Re-check your crawler access on a schedule so a silent block doesn't cost you months of AI visibility before anyone notices.

Takeaway

AI crawler access is a deliberate decision, not a default. Know the difference between training bots and search bots, never confuse Google-Extended with Googlebot, and audit your robots.txt regularly. The worst outcome is blocking the very crawlers that could be citing you, and never realizing it.