How to Block or Allow AI Crawlers (GPTBot, ClaudeBot, PerplexityBot) With robots.txt
AI labs have published their own named crawlers, separate from the search bots robots.txt was written for. Exactly which ones exist and what blocking does.
Key takeaways
- AI companies each publish a distinct crawler user-agent — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot — and robots.txt can allow or block each one independently.
- Blocking an AI crawler stops future crawling; it does not retroactively remove your content from a model already trained on older crawl data.
- Google-Extended controls Gemini and AI-features training use specifically — it's separate from the Googlebot rule that controls whether you appear in Search at all.
- robots.txt is a request, not an enforcement mechanism — reputable crawlers honor it as convention, but it can't technically stop one that ignores it.
- If the goal is GEO — being cited by these engines — the relevant crawler needs to be allowed, not blocked, or the site is removed from consideration entirely.
Why this suddenly became a real decision
robots.txt has existed since 1994, almost entirely to manage search-engine crawling. Since 2023, AI labs began publishing their own named, documented crawler user-agents — separate from Googlebot or Bingbot — specifically so site owners could decide, independently of search crawling, whether their content could be used for AI training or live AI retrieval. That split is new, and it's why the old "just block everything or allow everything" mental model no longer fits.
The AI crawlers actually worth naming individually
Each of these has its own exact user-agent string, and rules apply per bot — allowing or blocking one has no effect on the others.
- GPTBot — OpenAI's crawler used for training data collection
- ChatGPT-User — OpenAI's separate crawler for live browsing on a user's behalf inside ChatGPT, distinct from GPTBot
- ClaudeBot / anthropic-ai — Anthropic's crawlers
- PerplexityBot — Perplexity's crawler for live retrieval
- Google-Extended — controls Gemini and AI Overviews training use specifically, separate from the standard Googlebot rule
- CCBot — Common Crawl's crawler, whose archive many AI labs train from indirectly
- Applebot-Extended — controls Apple Intelligence training use
The tension: block for training, but allow for citation
If the actual goal is GEO — showing up as a cited source inside these tools' answers — blocking every AI crawler is self-defeating, since it removes the site from consideration entirely. The more deliberate move many sites make is blocking the pure training crawler (GPTBot, for instance) while still allowing the live-retrieval crawler for the same company (ChatGPT-User) that fetches a page in real time when a user's query points there.
A few real, working rule blocks
A robots.txt file allowing everything except GPTBot's training crawler:
- User-agent: GPTBot — Disallow: /
- User-agent: ChatGPT-User — Allow: /
- User-agent: * — Allow: /
What blocking doesn't do
Blocking a crawler today has no effect on data that crawler already collected in a previous pass — a model trained on an earlier crawl of the site may still reflect that content indefinitely. It's also purely voluntary: reputable AI labs document and honor their crawlers' robots.txt behavior, but the standard has no technical enforcement, so it can't stop a crawler — AI or otherwise — that chooses to ignore it.
Building the rules without hand-writing the syntax
Getting user-agent strings and directive syntax exactly right by hand is an easy place to introduce a silent mistake — a typo in a user-agent line simply gets ignored rather than throwing an error. The Robots.txt Generator builds correctly formatted rules for search crawlers and the named AI crawlers above without needing to memorize each exact string.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT's knowledge?
No — it only affects future crawling. A model already trained on an earlier crawl of your site may still reflect that older content.
If I block Google-Extended, do I disappear from Google Search?
No. Google-Extended only controls Gemini and AI-features training use. Standard Search visibility is controlled separately by the regular Googlebot rule.
Is robots.txt legally binding?
No — it's a voluntary standard. Reputable crawlers honor it as convention, not because they're legally required to.
Can I block one AI crawler but allow another?
Yes. Each AI crawler has its own distinct user-agent, so rules can be set independently per bot in the same robots.txt file.