Skip to content
Kuraib Ali

Why robots.txt silently blocks AI crawlers (and how to verify yours isn't)

robots.txt tells crawlers which parts of a site they may access. But "may access" means different things to different AI systems.

By Kuraib AliPublished Updated 13 min read

A black metal fence, representing the access rules robots.txt sets for crawlers, advisory rather than absolute
Photo: Unsplash

robots.txt is a text file that tells crawlers which parts of a site they may access. But "may access" doesn't mean the same thing to every AI system. Some AI bots honour it without exception. OpenAI's own documentation says robots.txt "may not apply" to one of its three crawlers. And two of the biggest names in this space, Google-Extended and Applebot-Extended, aren't crawlers at all; they're permission flags on content Googlebot and Applebot already fetched. Getting this precise matters more than copying a template file.

What is robots.txt?

A robots.txt file lives at the root of a site and tells crawlers which URLs they can access. Google says it is used mainly to avoid overloading a site with requests, and that it is not a mechanism for keeping a page out of Google. To keep a page out of Google, use noindex or password protection.

What robots.txt does (and doesn't)

robots.txt is a request, not a lock. It sits at a site's root and tells well-behaved crawlers which paths they're welcome to fetch, grouped by User-agent. A crawler that respects the standard will read it before crawling and skip whatever's marked Disallow.

What it doesn't do: hide anything. A disallowed URL is still publicly reachable by anyone who has the link. The file itself is public, and blocking a path in it doesn't password-protect or encrypt what's there. It only changes whether compliant automated crawlers proactively fetch it. That distinction matters for everything that follows.

Which AI crawlers read robots.txt, and which don't quite

Verified directly against each operator's own current documentation.

OperatorBotsRespects robots.txt?
OpenAIGPTBot (training), OAI-SearchBot (search index), ChatGPT-User (live user fetch)GPTBot and OAI-SearchBot: yes. ChatGPT-User: OpenAI's own docs say rules "may not apply," since it's user-triggered, not automated crawling.
AnthropicClaudeBot (training), Claude-SearchBot (search index), Claude-User (live user fetch)All three. Anthropic's documentation makes no exception for Claude-User.
PerplexityPerplexityBot (index), Perplexity-User (retrieval)PerplexityBot: declared as respecting it. Perplexity-User: Perplexity's own docs say it "generally ignores robots.txt rules," since a user requested the fetch. Separately, a documented 2025 investigation found undeclared, rotating crawlers accessing sites that had blocked the declared bots. Perplexity disputed the finding.

Each bot within an operator is independently controllable. Allowing OAI-SearchBot while disallowing GPTBot is explicitly supported and documented, and the same independence holds for Anthropic's three bots. Being allowed in robots.txt is only the access half of the equation. See do AI crawlers read your site for what each of these bots does once it's let in, including which ones render JavaScript.

Writing a robots.txt that correctly handles every named AI crawler

Most template robots.txt files circulating online cover two or three bots and call it done. A file that accounts for today's AI crawlers needs its own block per bot, because "block AI" isn't one decision, it's several, and each operator's own documentation is explicit that its bots are independently controllable.

Beyond the OpenAI, Anthropic, and Perplexity bots already covered above, a handful of other named crawlers show up regularly in server logs and are worth naming precisely rather than lumping into a generic "AI bot" bucket:

OperatorBotUser-agent tokenPurposeRespects robots.txt?
MetaMeta-ExternalAgentmeta-externalagentCrawls the web to train Meta's foundation AI models and to index content for Meta's products, per Meta's own developer documentationYes
Common CrawlCCBotCCBotBuilds the open, publicly available web-crawl dataset that many AI labs train on, per Common Crawl's own documentationYes
AmazonAmazonbotAmazonbotImproves Amazon's products and services, including crawling for AI-related features, per Amazon's own developer documentationYes
AmazonAmzn-SearchBotAmzn-SearchBotImproves Amazon search experiences; Amazon's own documentation states it does not crawl content for generative AI model trainingYes

A robots.txt block that deliberately covers every operator named here, with each bot addressed on its own line rather than folded into a wildcard, looks like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Amazonbot
Allow: /

User-agent: Amzn-SearchBot
Allow: /

That specific example opts every operator's model-training crawler out (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot) while leaving each operator's search-indexing and live-user-fetch bots alone, on the reasoning that training use and answer-surfacing use are different decisions. It's a starting point to edit, not a rule to copy verbatim. The right answer depends on whether the goal is opting out of training data, staying out of AI answers entirely, or something narrower, like keeping one directory (a staging area, an internal wiki, a paywalled section) out of everything. A Disallow: /some-path scoped to one directory works the same way under any of these User-agent groups as a full-site Disallow: / does.

Two syntax details that change what a rule matches: a Disallow value is a path prefix match, not a pattern, so Disallow: /blog also blocks /blog-archive unless a trailing slash is added; and when a bot matches more than one group (a specific User-agent: GPTBot block plus a generic User-agent: * block), the specific group wins for that bot and the wildcard group is ignored for it entirely, per the robots.txt standard (RFC 9309). That's the mechanism behind the wildcard mistake in the next section.

Both details in one short file:

robots.txtExample
  1. User-agent: *
  2. Disallow: /blogA prefix match: /blog-archive is blocked too
  3. Disallow: /staging/
  4.  
  5. User-agent: GPTBotGPTBot follows this group and ignores the * group entirely
  6. Disallow: /Blocks GPTBot only; OAI-SearchBot still falls back to the * group

Google-Extended and Applebot-Extended aren't crawlers

This is the part most guides get imprecise, and it's the same structure for both companies once you see it: neither Google-Extended nor Applebot-Extended is a crawling bot. Googlebot and Applebot do the fetching, as they always have. Google-Extended and Applebot-Extended are permission tokens: rules that govern what may be done afterwards with content those crawlers already collected.

Google-Extended controls whether that already-crawled content can be used to train future Gemini models and to "ground" answers in Gemini Apps and Vertex AI. Google's own documentation states it directly: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Disallowing it doesn't touch indexing, ranking, AI Overviews, or AI Mode. Those run on ordinary Googlebot plus separate controls (nosnippet, data-nosnippet, max-snippet, noindex). Google also shipped a separate control for AI Overviews and AI Mode: a toggle inside Search Console, announced on Google's own blog on 3 June 2026 and rolled out to all sites by 31 August 2026. It lets a site opt out of AI Overviews, AI Mode and AI Overviews in Discover without affecting its regular Search rankings or its snippet eligibility elsewhere. It lives in Search Console settings, not in robots.txt, and it's a separate lever again from both Google-Extended and the nosnippet family. See how to show up in AI search results for what influences inclusion in those surfaces.

Applebot-Extended works the same way. Apple's own documentation is explicit: "Applebot-Extended does not crawl webpages." It only governs whether data already collected by Applebot may train Apple's generative AI features (Apple Intelligence, Siri, and related products). Disallowing it doesn't remove a page from Spotlight, Siri, or Safari search results. Those stay governed by the regular Applebot rule.

Practical result: a robots.txt that disallows Google-Extended and Applebot-Extended alongside Googlebot and Applebot is opting out of two different things at once, so be clear which one you meant to change.

A few common mistakes

  • A wildcard block catches everything, including bots you meant to allow. User-agent: * followed by Disallow: / blocks every crawler that doesn't have its own more specific rule elsewhere in the file. A leftover from a staging environment is the most common way this happens by accident.
  • Blocking the wrong bot for the goal. Disallowing GPTBot stops training use but leaves OAI-SearchBot and ChatGPT-User untouched. If the goal was full opt-out, all three need their own rule.
  • Duplicate or conflicting user-agent groups. Two separate blocks for the same bot with different rules creates ambiguity about which one applies.

A newer mistake: assuming a permissive robots.txt means a bot gets through. A growing number of sites also block AI crawlers at the content delivery network (CDN) or firewall layer. That decision lives outside robots.txt and doesn't show up when you read the file. Cloudflare announced, on 1 July 2026, a reclassification of AI traffic into three categories, Search, Agent, and Training, and confirmed that starting 15 September 2026, newly onboarded domains get Training and Agent traffic blocked by default on any page that carries advertising, while Search traffic stays allowed by default. This sits on a separate layer from a site's own robots.txt: robots.txt asks well-behaved bots to stay out, while a network-level control enforces the decision whether a bot asks or not. A site can have a fully permissive robots.txt and still block GPTBot or ClaudeBot at the edge, or the reverse, a Disallow-everything robots.txt with a CDN dashboard set to allow everything through. Checking one without the other tells half the story, which is why the log-file check below matters more than reading the file alone.

A golden key resting on a wooden surface, representing the permission tokens Google-Extended and Applebot-Extended control

Could something on your site already be blocking AI bots?

WordPress itself defaults to permissive. Its built-in virtual robots.txt allows all user agents unless something changes it. But some SEO plugins now ship an AI-bot-blocking feature as an option. Yoast SEO Premium, for one, has a toggle that adds Disallow rules for GPTBot and Google's Gemini/Vertex bots, confirmed directly from Yoast's own documentation as something the site owner switches on, not something enabled by default.

Check rather than assume: if visibility has dropped and nobody remembers changing robots.txt directly, a toggle switched on by a previous developer, during a migration, or during a plugin update is a more likely culprit than the plugin quietly blocking bots on its own.

How to check your robots.txt, step by step

  1. Read the live file directly at yoursite.com/robots.txt, not a cached version, not what a plugin's settings page claims it generates.
  2. Find every User-agent group and note exactly what each one disallows.
  3. Cross-check each AI-related rule against what that operator's own documentation says the bot is for, not a generic list, since the same name can mean different things (Google-Extended being the clearest example).
  4. If a change was just made, allow for propagation lag before assuming it hasn't worked, since crawlers don't re-read robots.txt on every request.
  5. Remember robots.txt is advisory. Compliant bots follow it; not every bot is guaranteed to be one, which is why sensitive content needs real access control, not just a Disallow line.
  6. For proof rather than inference, check server logs directly for requests from GPTBot, ClaudeBot, PerplexityBot, and the rest. The same log-file method covered in crawl budget and log files confirms whether a bot is visiting, not just theoretically allowed to.

Green text scrolling on a dark terminal screen, representing the server log entries that confirm whether a bot visited

How to verify a crawler's identity server-side

Reading a User-agent string out of a log line proves nothing on its own. It's a value the client sends, and anyone can send it: a scraper claiming to be GPTBot gets whatever access the real GPTBot would get if the check stops at the header. Every operator covered here that publishes verification guidance points to the same two methods, used together rather than either alone.

Matching against published IP ranges. OpenAI publishes a separate, machine-readable JSON file per bot rather than one shared list, specifically so a check can be pinned to one crawler: openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, and openai.com/adsbot.json. Common Crawl publishes IPv4 and IPv6 ranges for CCBot as JSON at index.commoncrawl.org/ccbot.json. Amazon publishes Amazonbot's ranges at developer.amazon.com/amazonbot/ip-addresses. Anthropic publishes an address list too, at claude.com/crawling/bots.json, but with a caveat, in its own words: Anthropic's documentation warns that blocking by IP address "may not work correctly or persistently guarantee an opt-out," and gives its reason: blocking by IP "impedes our ability to read your robots.txt file."

Forward-confirmed reverse DNS (FCrDNS). This is the sturdier check where it's available, and it's the same method Google has long recommended for verifying Googlebot. It's a four-step DNS (domain name system) lookup: reverse-DNS the request's source IP to get a hostname, confirm that hostname ends in the operator's own domain, forward-DNS that hostname back to an IP address, and confirm it matches the original IP. The forward step is what defeats spoofing: an attacker can point a fake reverse-DNS record at their own IP, but can't forge the operator's authoritative forward DNS answer for a domain they don't control. Common Crawl documents this directly for CCBot, whose dedicated IPs resolve to hostnames like 18-97-14-84.crawl.commoncrawl.org, letting a site confirm a request against Common Crawl's own domain rather than trusting the header.

A glowing fingerprint on a dark scanner surface, representing the forward-confirmed reverse DNS check that verifies a crawler's identity beyond its user-agent string

Neither method is instant or free of caveats. IP ranges change, so the practical guidance from every operator that publishes them is the same: fetch the current file at request time or on a short cache, rather than hardcoding a list into a firewall rule once and forgetting it. And an IP that doesn't appear in a published range isn't proof of a fake crawler either, since legitimate user-triggered fetches (ChatGPT-User, Claude-User, Perplexity-User) can originate from infrastructure the published training-bot ranges don't cover. Treat a failed IP or FCrDNS check as a reason to look closer, not as automatic grounds to block, unless the traffic pattern itself (request volume, paths hit, timing) already looks abusive.

robots.txt is not a security boundary

Disallow doesn't mean private. The file itself is public, anyone can read it, and it only asks well-behaved automated crawlers not to fetch a path. It can't stop a direct visit, and it can't guarantee every crawler will honour it. Anything that needs to stay confidential belongs behind real authentication, not a line in a text file.