Your robots.txt is a short text file at yoursite.com/robots.txt that tells bots which parts of your site they may visit. For years it mostly spoke to Googlebot. Today it speaks to dozens of bots, and most AI companies run several of them, each with a different job.
That’s where sites go wrong. A shop that wants to keep its content out of AI training blocks a bot that actually builds search results, and disappears from AI answers. Another never touched its file, but its CDN added rules on top. This post sorts the bots by what they do, so you can see what your file is really saying.
What robots.txt is, and what it isn’t
The rules are a request, not a lock. The standard says so in as many words: “These rules are not a form of access authorization” (RFC 9309). Well-behaved bots follow them. Anything that ignores them can still read your public pages.
Not having a file is fine. Google treats a missing file “as if a valid robots.txt file didn’t exist” and crawls without restrictions (Google). A broken one is worse: if your server answers the robots.txt request with an error, Google stops crawling the whole site for the first 12 hours while it keeps retrying (same page).
Three kinds of bots
Almost every AI bot falls into one of three groups. What a block does depends on the group, not on the company.
Training bots collect pages to train AI models. Blocking them keeps your content out of future training. It doesn’t take you out of search or out of AI answers.
- GPTBot, OpenAI’s training bot (OpenAI).
- ClaudeBot, Anthropic’s (Anthropic).
- Google-Extended, which controls Gemini training and isn’t a separate crawler. Google says it “does not impact a site’s inclusion in Google Search” (Google).
- Applebot-Extended, Apple’s training opt-out. Pages that block it “can still be included in search results” (Apple).
- Meta-ExternalAgent, which crawls “for use cases such as training foundation AI models” (Meta).
- CCBot, Common Crawl’s crawler, which builds an open archive of the web that others build on (Common Crawl).
Search bots build the index that an engine or assistant searches when someone asks a question. Blocking them takes you out of those answers. OpenAI says sites that block OAI-SearchBot “will not be shown in ChatGPT search answers” (OpenAI).
- Googlebot and bingbot, which also feed Google’s and Microsoft’s AI answers.
- OAI-SearchBot for ChatGPT search.
- Claude-SearchBot for Claude’s search (Anthropic).
- PerplexityBot for Perplexity (Perplexity).
- Applebot for Siri, Spotlight and Safari (Apple).
- Meta-WebIndexer, which helps Meta AI “cite and link to your content” (Meta).
- Amzn-SearchBot for search in Amazon’s products, including Alexa (Amazon).
Fetchers visit a page because a person asked, for example when someone pastes your link into a chat. Because a person started the request, most of them treat your robots.txt as advice at best.
- ChatGPT-User: “robots.txt rules may not apply” (OpenAI).
- Perplexity-User “generally ignores robots.txt rules” (Perplexity).
- Meta-ExternalFetcher “may bypass robots.txt rules” (Meta).
- Amzn-User “may not follow all robots.txt directives” (Amazon).
- Google’s user-triggered fetchers “generally ignore robots.txt rules” (Google).
- Claude-User is the exception: Anthropic’s bots honour robots.txt, and blocking Claude-User “prevents our system from retrieving your content in response to a user query” (Anthropic).
The mistakes that cost the most
Blocking everyone by accident. Disallow: / under User-agent: * shuts out every bot that follows the rules, Google included. It’s often left over from a staging site or a launch in a hurry.
Blocking search to stop training. If the goal is to stay out of training, block the training bots by name. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from those assistants’ answers and doesn’t stop training at all.
Rules you never wrote. CDNs can block AI bots for you, and Cloudflare can add its own block of rules to your file: it will “prepend our managed robots.txt before your existing robots.txt, combining both into a single response” (Cloudflare). Your own file can look fine while the one bots receive says something else. Always check the live address, not the file on your server.
Expecting a block to stop agents. As the fetchers above show, a person’s assistant may visit your page whatever your file says. If a page must stay private, protect it with a login, not with robots.txt.
How to see what bots really get
Open yoursite.com/robots.txt in a private browser window. That’s the file bots receive, including anything your CDN added on top. Read it from the top: every User-agent line starts a group, and the Disallow lines under it apply to the bots that group names. A bot follows the group that names it most specifically, and falls back to the * group only if nothing names it.
Then open Google Search Console and find the robots.txt report. It “shows which robots.txt files Google found for the top 20 hosts on your site”, when Google last fetched each one, whether the fetch worked, and any rules it couldn’t read (Google). A failed fetch there is worth fixing first, for the 12-hour reason above.
A sensible starting point
Our view: let search bots and fetchers in, because they’re how assistants find and quote you, and make the training decision separately, on its own merits. A file that does that can be this short:
User-agent: *
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
Sitemap: https://yoursite.com/sitemap.xmlThe first group lets every bot in. The second blocks only the training bots; delete it if you’re happy for your content to be used in training. Several User-agent lines can share one group of rules, as the standard allows (RFC 9309). The last line points bots that read sitemaps, Google and Bing, at your list of pages.
Whatever you decide, write it down in the file on purpose. The worst robots.txt is the one nobody chose.
