How to Block AI Training but Allow AI Search in robots.txt
Short answer: Give each training crawler its own User-agent group with Disallow: /, and leave search crawlers allowed. AI companies use different crawlers for training and for search: for example GPTBot trains OpenAI's models while OAI-SearchBot finds pages for ChatGPT search. Blocking the first does not block the second.
Which AI crawlers train models and which power AI search?
Each AI company documents its own crawler names. This is the split we use in SEO AI Assistant, based on what the providers publish:
| Company | Search or answer crawlers (allow to be cited) | Training crawlers (safe to block) |
|---|---|---|
| OpenAI | OAI-SearchBot, ChatGPT-User | GPTBot |
| Anthropic | Claude-SearchBot, Claude-User | ClaudeBot |
| Perplexity | PerplexityBot, Perplexity-User | none listed |
| Googlebot (regular Search) | Google-Extended | |
| Apple | Applebot | Applebot-Extended |
| Meta | Meta-ExternalFetcher | Meta-ExternalAgent |
| Others | DuckAssistBot, MistralAI-User | CCBot (Common Crawl), Amazonbot, Bytespider |
Two details matter. OpenAI's crawler documentation says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, while GPTBot only covers training. Google's crawler documentation says Google-Extended does not affect whether a site is included in Google Search.
Copy-ready robots.txt example
This file blocks the training crawlers above and allows everything else, including all search crawlers. Replace the sitemap address with yours:
# Block AI model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Bytespider
Disallow: /
# Everyone else, including AI search crawlers
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlA crawler follows the group that names it and ignores the * group, so OAI-SearchBot, PerplexityBot and the rest fall under User-agent: * here and stay allowed.
How to set it up in 4 steps
- Open your current file at
yoursite.com/robots.txtand keep any rules you already rely on, such as blocked admin paths. - Add one group per training crawler as in the example, above your
User-agent: *group. - Check the
*group doesn't containDisallow: /, which would block every search crawler not named elsewhere. - Check your CDN or host for a separate AI-bot blocking setting, which can block crawlers before robots.txt is even read.
Mistakes that block AI search by accident
- Blocking everything with
User-agent: *andDisallow: /to stop AI training. It also blocks search crawlers, and Google. - Blocking OAI-SearchBot or PerplexityBot because they look like "AI bots". They are the ones that cite you.
- Turning on a CDN's blanket AI-bot block without checking which crawlers it covers.
- Forgetting robots.txt is a request. Well-behaved crawlers follow it, but it is not access control. Use authentication for anything private.
See which AI crawlers your site blocks
SEO AI Assistant is a free Chrome extension that reads a site's robots.txt and checks it against 10 AI search crawlers. Blocking one fails the Allow AI crawlers check. Blocked training crawlers are listed but cost no points, because blocking model training is your choice. Every rule is explained on How scores work.
AI crawler questions
Will blocking Google-Extended hurt my Google rankings?
No. Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls whether your content is used for Google's AI models.
Should I block GPTBot?
Only if you don't want OpenAI to use your pages for model training. Blocking GPTBot doesn't remove you from ChatGPT search, which uses OAI-SearchBot.
Do AI crawlers obey robots.txt?
The major AI companies say their documented crawlers follow robots.txt. Some fetchers that act on a user's direct request may behave differently, and unknown scrapers may ignore it, so robots.txt is not a security tool.
How do I check if my site is blocking AI crawlers?
Open yoursite.com/robots.txt and look for groups naming AI crawlers or a User-agent: * group with Disallow: /. A free checker such as SEO AI Assistant lists every AI search and training crawler your robots.txt blocks.
Do I need a separate file for AI crawlers?
No. AI crawlers read the same robots.txt as search engines. An llms.txt file is a different, optional summary file and does not control access.