Why robots.txt Silently Blocks AI Crawlers (and How to Verify Yours Isn't)

robots.txt is a text file that tells crawlers which parts of a site they may access. But "may access" doesn't mean the same thing to every AI system. Some AI bots honor it without exception. OpenAI's own documentation says robots.txt "may not apply" to one of its three crawlers. And two of the biggest names in this space, Google-Extended and Applebot-Extended, aren't crawlers at all; they're permission flags on content Googlebot and Applebot already fetched. Getting this precise matters more than copying a template file.

What robots.txt actually does (and doesn't)

robots.txt is a request, not a lock. It sits at a site's root and tells well-behaved crawlers which paths they're welcome to fetch, grouped by User-agent. A crawler that respects the standard will read it before crawling and skip whatever's marked Disallow.

What it doesn't do: hide anything. A disallowed URL is still publicly reachable by anyone who has the link. The file itself is public, and blocking a path in it doesn't password-protect or encrypt what's there. It only changes whether compliant automated crawlers proactively fetch it. That distinction matters for everything that follows.

Which AI crawlers read robots.txt, and which don't quite

Verified directly against each operator's own current documentation.

Each bot within an operator is independently controllable. Allowing OAI-SearchBot while disallowing GPTBot is explicitly supported and documented, and the same independence holds for Anthropic's three bots. Being allowed in robots.txt is only the access half of the equation. See do AI crawlers actually read your site for what each of these bots does once it's let in, including which ones render JavaScript.

Writing a robots.txt that correctly handles every named AI crawler

Most template robots.txt files circulating online cover two or three bots and call it done. A file that actually accounts for the current AI crawler landscape needs its own block per bot, because "block AI" isn't one decision, it's several, and each operator's own documentation is explicit that its bots are independently controllable.

Beyond the OpenAI, Anthropic, and Perplexity bots already covered above, a handful of other named crawlers show up regularly in server logs and are worth naming precisely rather than lumping into a generic "AI bot" bucket:

A robots.txt block that deliberately covers every operator named in this guide, with each bot addressed on its own line rather than folded into a wildcard, looks like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Amazonbot
Allow: /

User-agent: Amzn-SearchBot
Allow: /

That specific example opts every operator's model-training crawler out (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot) while leaving each operator's search-indexing and live-user-fetch bots alone, on the reasoning that training use and answer-surfacing use are genuinely different decisions. It's a starting point to edit, not a rule to copy verbatim: the right answer depends on whether the actual goal is opting out of training data, staying out of AI answer surfaces entirely, or something narrower like keeping a specific directory (a staging area, an internal wiki, a paywalled section) out of everything. A Disallow: /some-path scoped to one directory works the same way under any of these User-agent groups as a full-site Disallow: / does.

Two syntax details that change what a rule actually matches: a Disallow value is a path prefix match, not a pattern, so Disallow: /blog also blocks /blog-archive unless a trailing slash is added; and when a bot matches more than one group (a specific User-agent: GPTBot block plus a generic User-agent: * block), the specific group wins for that bot and the wildcard group is ignored for it entirely, per the robots.txt standard (RFC 9309). That's the mechanism behind the wildcard mistake in the next section.

Google-Extended and Applebot-Extended aren't crawlers

This is the part most guides get imprecise, and it's the same structure for both companies once you see it: neither Google-Extended nor Applebot-Extended is a crawling bot. Googlebot and Applebot do the actual fetching, exactly as they always have. Google-Extended and Applebot-Extended are permission tokens: rules that govern what may be done afterward with content those crawlers already collected.

Google-Extended controls whether that already-crawled content can be used to train future Gemini models and to "ground" answers in Gemini Apps and Vertex AI. Google's own documentation states it directly: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Disallowing it doesn't touch indexing, ranking, AI Overviews, or AI Mode. Those run on ordinary Googlebot plus separate controls (nosnippet, data-nosnippet, max-snippet, noindex). Google also shipped a genuinely separate control for AI Overviews and AI Mode specifically: a toggle inside Search Console, announced on Google's own blog on June 3, 2026 and rolled out to all sites worldwide by August 31, 2026, that lets a site opt out of appearing in AI Overviews, AI Mode, and AI Overviews in Discover without affecting its regular Search rankings or its snippet eligibility elsewhere. It lives in Search Console settings, not in robots.txt, and it's a separate lever again from both Google-Extended and the nosnippet family. See how to show up in AI search results for what actually influences inclusion in those surfaces.

Applebot-Extended works the same way. Apple's own documentation is explicit: "Applebot-Extended does not crawl webpages." It only governs whether data already collected by Applebot may train Apple's generative AI features (Apple Intelligence, Siri, and related products). Disallowing it doesn't remove a page from Spotlight, Siri, or Safari search results. Those stay governed by the regular Applebot rule.

Practical result: a robots.txt that disallows Google-Extended and Applebot-Extended alongside Googlebot and Applebot is opting out of two different things at once, and it's worth knowing which one you actually meant to change.

A few common mistakes

• A wildcard block catches everything, including bots you meant to allow. User-agent: * followed by Disallow: / blocks every crawler that doesn't have its own more specific rule elsewhere in the file. A leftover from a staging environment is the most common way this happens by accident.
• Blocking the wrong bot for the goal. Disallowing GPTBot stops training use but leaves OAI-SearchBot and ChatGPT-User untouched. If the actual goal was full opt-out, all three need their own rule.
• Duplicate or conflicting user-agent groups. Two separate blocks for the same bot with different rules creates ambiguity about which one applies.

A newer mistake worth naming specifically: a permissive robots.txt no longer guarantees a bot actually gets through, because a growing number of sites now also block AI crawlers at the CDN or firewall layer, a decision that lives entirely outside robots.txt and doesn't show up when reading the file. Cloudflare's own blog announced, on July 1, 2026, a reclassification of AI traffic into three categories, Search, Agent, and Training, and confirmed that starting September 15, 2026, newly onboarded domains (plus existing free-plan sites) get Training and Agent traffic blocked by default on any page that carries advertising, while Search traffic stays allowed by default. Cloudflare's post is explicit that this sits on a separate layer from a site's own robots.txt: "a robots.txt file tells well-behaved bots... what they may access," while the network-level control "enforces that decision against bots that never ask." A site can have a fully permissive robots.txt and still block GPTBot or ClaudeBot at the edge, or the reverse, a Disallow-everything robots.txt with a CDN dashboard set to allow everything through. Checking one without checking the other only tells half the story, which is exactly why the log-file check later in this guide matters more than reading the file alone.

Could something on your site already be blocking AI bots?

WordPress itself defaults to permissive. Its built-in virtual robots.txt allows all user agents unless something changes it. But some SEO plugins now ship an AI-bot-blocking feature as an option. Yoast SEO Premium, for one, has a toggle that adds Disallow rules for GPTBot and Google's Gemini/Vertex bots, confirmed directly from Yoast's own documentation as something the site owner switches on, not something enabled by default.

Worth checking rather than assuming either way: if visibility has dropped and nobody remembers changing robots.txt directly, a toggle switched on by a previous developer, during a migration, or during a plugin update is a more likely culprit than the plugin quietly blocking bots on its own.

How to actually check

1. Read the live file directly at yoursite.com/robots.txt, not a cached version, not what a plugin's settings page claims it generates.
2. Find every User-agent group and note exactly what each one disallows.
3. Cross-check each AI-related rule against what that operator's own documentation says the bot is actually for, not a generic list, since the same name can mean different things (Google-Extended being the clearest example).
4. If a change was just made, allow for propagation lag before assuming it hasn't worked. OpenAI's own documentation notes roughly 24 hours for search-related changes to take effect.
5. Remember robots.txt is advisory. Compliant bots follow it; not every bot is guaranteed to be one, which is why sensitive content needs real access control, not just a Disallow line.
6. For proof rather than inference, check server logs directly for requests from GPTBot, ClaudeBot, PerplexityBot, and the rest. The same log-file method covered in crawl budget and log files confirms whether a bot is actually visiting, not just theoretically allowed to.

How to verify a crawler's identity server-side

Reading a User-agent string out of a log line proves nothing on its own. It's a value the client sends, and anyone can send it: a scraper claiming to be GPTBot gets whatever access the real GPTBot would get if the check stops at the header. Every operator covered in this guide that publishes verification guidance points to the same two methods, used together rather than either alone.

Matching against published IP ranges. OpenAI publishes a separate, machine-readable JSON file per bot rather than one shared list, specifically so a check can be pinned to one crawler: openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, and openai.com/adsbot.json. Common Crawl publishes IPv4 and IPv6 ranges for CCBot as JSON at index.commoncrawl.org/ccbot.json. Amazon publishes Amazonbot's ranges at developer.amazon.com/amazonbot/ip-addresses. Anthropic publishes an address list too, at claude.com/crawling/bots.json, but with a caveat worth repeating in its own words: Anthropic's documentation warns that blocking by IP address "may not work correctly or persistently guarantee an opt-out," because the published ranges sit inside shared cloud infrastructure rather than an Anthropic-only network, so an IP match there is weaker evidence than an IP match against OpenAI's or Common Crawl's dedicated ranges.

Forward-confirmed reverse DNS (FCrDNS). This is the sturdier check where it's available, and it's the same method Google has long recommended for verifying Googlebot. It's a four-step lookup: reverse-DNS the request's source IP to get a hostname, confirm that hostname ends in the operator's own domain, forward-DNS that hostname back to an IP address, and confirm it matches the original IP. The forward step is what actually defeats spoofing: an attacker can point a fake reverse-DNS record at their own IP, but can't forge the operator's authoritative forward DNS answer for a domain they don't control. Common Crawl documents this directly for CCBot, whose dedicated IPs resolve to hostnames like 18-97-14-84.crawl.commoncrawl.org, letting a site confirm a request against Common Crawl's own domain rather than trusting the header.

Neither method is instant or free of caveats. IP ranges change, so the practical guidance from every operator that publishes them is the same: fetch the current file at request time or on a short cache, rather than hardcoding a list into a firewall rule once and forgetting it. And an IP that doesn't appear in a published range isn't proof of a fake crawler either, since legitimate user-triggered fetches (ChatGPT-User, Claude-User, Perplexity-User) can originate from infrastructure the published training-bot ranges don't cover. Treat a failed IP or FCrDNS check as a reason to look closer, not as automatic grounds to block, unless the traffic pattern itself (request volume, paths hit, timing) already looks abusive.

robots.txt is not a security boundary

Disallow doesn't mean private. The file itself is public, anyone can read it, and it only asks well-behaved automated crawlers not to fetch a path. It can't stop a direct visit, and it can't guarantee every crawler will honor it. Anything that actually needs to stay confidential belongs behind real authentication, not a line in a text file.