Skip to content
Kuraib Ali

Do AI crawlers read your site?

GPTBot, ClaudeBot and PerplexityBot split into three different jobs, and most can't see JavaScript-rendered content at all.

By Kuraib AliPublished Updated 11 min read

Water droplets on a spider web against a dark background, representing the web of AI crawlers that fetch a site for different reasons
Photo: Unsplash

Most of them, no. Not the way a browser does. The AI crawlers behind ChatGPT, Claude, and Perplexity split into three different jobs with three different sets of rules, and independent testing on real production traffic found that none of the major ones render JavaScript. If your content only appears after a script runs, it's invisible to almost every AI system that isn't Google.

Not one crawler: three jobs

"AI crawler" gets used as if it's one thing. It isn't. Every major AI company now runs multiple bots, split across three distinct jobs, and the job determines what blocking it does to you:

  1. Training crawlers collect content to train future models. Blocking these opts you out of a model's underlying knowledge. It does nothing to your visibility today.
  2. Search-index crawlers build the index a chatbot's search feature draws from. Blocking these is the one that costs you citations.
  3. User-triggered fetchers run when a real person asks the assistant to open or check a specific page. Several of these don't reliably respect robots.txt at all, because the request is initiated by a person, not an automated crawl.

Treating all three as "the AI bots" and blanket-blocking them is the single most common mistake site owners make, and it usually blocks the wrong one.

Who's out there

Verified against each company's own current developer documentation.

CompanyBotJobNotes
OpenAIOAI-SearchBotSearch indexCan be allowed independently of GPTBot. Disallowing GPTBot alone does not remove you from ChatGPT search.
GPTBotTraining
ChatGPT-UserUser-triggeredOpenAI's own developer docs state it directly: ChatGPT-User "is not used for crawling the web in an automatic fashion," and because the action is user-initiated, "robots.txt rules may not apply."
OAI-AdsBotAd-page validation
AnthropicClaudeBotTraining
Claude-UserUser-triggeredNotably, Anthropic states all three of its bots honour robots.txt, including this one, an exception among the major vendors.
Claude-SearchBotSearch index
PerplexityPerplexityBotSearch indexPerplexity's own help centre says a disallowed page can still surface as a domain, headline, and short summary.
Perplexity-UserUser-triggeredPerplexity's own docs say it "generally ignores robots.txt rules," because a user requested the fetch. Separately, Cloudflare's 2025 investigation reported undeclared Perplexity crawling in general, not this agent specifically, getting around blocks. robots.txt is not a reliable stop here.
GoogleGoogle-ExtendedAI use onlyControls whether content is used for Gemini apps and Vertex AI. Blocking it does not affect Google Search, including AI Overviews and AI Mode.
GooglebotSearch + AI OverviewsThe same crawl feeds AI Overviews and AI Mode. There's no separate "AI Overviews bot."
AppleApplebot-ExtendedTraining opt-outApple's own support page states this token "does not crawl webpages," it only governs how already-collected Applebot data is used.
OthersAmazonbot, Meta-ExternalAgent, CCBot, BytespiderMixedBytespider (ByteDance) doesn't appear to have an official documentation page, and has a poor compliance reputation. Verify any claimed visit by IP, not by user-agent string alone.

Do they render JavaScript?

Mostly, no. Analysis of hundreds of millions of real AI-crawler requests (Vercel and MERJ, December 2024) found no evidence of JavaScript execution from GPTBot, OAI-SearchBot, ChatGPT-User, or ClaudeBot. They sometimes download a page's JavaScript files (GPTBot in roughly 11% of requests, ClaudeBot in roughly 24%), but they read the file as text and never run it.

Google is the exception: Googlebot's crawl-render-index pipeline does execute JavaScript, which is why a client-rendered page can rank normally on Google while showing up as a near-blank page to every other AI crawler. If your headline, body copy, or key facts only appear after a script runs, server-side rendering, static generation, or prerendering is what fixes that, not a meta tag or a plugin setting.

Close-up of a white robotic arm, representing the automated, non-human nature of AI crawler traffic

The robots.txt mistake most sites make

Three patterns account for most of the "why isn't my site showing up in ChatGPT" cases:

  • Blocking search bots by accident. A rule meant to stop AI training (disallowing GPTBot, ClaudeBot) gets copy-pasted broadly and catches OAI-SearchBot or Claude-SearchBot too, the ones that feed citations.
  • A firewall blocking what robots.txt allows. Web application firewall (WAF) and rate-limiting rules (Cloudflare, and similar tools from other providers) can return 429 or 403 responses to a bot your robots.txt technically permits. The bot can't solve a CAPTCHA, so the block is silent and permanent from its side.
  • Assuming a disallow stops everything. It doesn't reliably stop ChatGPT-User or Perplexity-User, since those fire on a real person's request rather than an automated crawl.

The fix isn't a blanket allow or a blanket block. It's deciding your policy per job. Allow the search/retrieval bots if AI visibility matters to you. Decide separately whether you want your content used for training.

How to test whether a specific AI crawler can reach a page

Reading robots.txt only shows what's supposed to happen. Testing shows what happens at the server or content delivery network (CDN), and the two disagree more often than site owners expect, usually because of a firewall rule nobody connected to AI crawlers in the first place.

  1. Fetch the page with the crawler's real user-agent string. curl -A "<full UA string>" https://yoursite.com/page -I sends a request identifying as that bot and returns only the response headers. OpenAI's current developer docs publish the exact strings: GPTBot identifies as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot, with separate strings for OAI-SearchBot, ChatGPT-User, and OAI-AdsBot. Anthropic doesn't publish a full user-agent string for ClaudeBot; the form usually observed in server logs ends ClaudeBot/1.0; +claudebot@anthropic.com. A 200 response means nothing at the server layer is actively blocking that identity. A 403, a 429, or a redirect into a challenge page means something is, regardless of what robots.txt says.
  2. Don't stop at robots.txt when a block shows up. A page can be fully allowed in robots.txt and still return a non-200 to the same user agent, because a WAF rule, a rate limiter, or a bot-management product is filtering the request before robots.txt is ever consulted. That's a firewall problem, not a robots.txt problem, and it needs fixing in the firewall.
  3. Allow for propagation lag before concluding a fix didn't work. Crawlers don't re-read robots.txt on every request, so a change takes time to reach them. Retesting five minutes after an edit and assuming it failed is a common false negative.
  4. Treat the curl test as proof of server behaviour, not proof of who visited. Anyone can send a request with GPTBot's exact user-agent string; the curl test only confirms what your server does when it sees that string. Confirming the real bot showed up requires checking access logs against the vendor's published IP ranges, covered next: OpenAI publishes per-bot JSON files (openai.com/gptbot.json, openai.com/searchbot.json, and similarly named files for its other bots), and Anthropic publishes one combined list covering ClaudeBot, Claude-User, and Claude-SearchBot at claude.com/crawling/bots.json.

Run through those four steps per crawler that matters to you, rather than assuming that because one AI bot gets through, they all do. A rule scoped to GPTBot in a WAF product doesn't automatically catch ClaudeBot or PerplexityBot, and a security team that added AI-bot rules months ago may not have kept the list current as new bots launched.

A traffic light showing a green signal against the sky, representing the pass or block verdict a direct crawler test reveals

How to check your own logs

A user-agent string alone proves nothing. It's a text field anyone can fake. Google's own documentation describes the reliable method, forward-confirmed reverse DNS: look up the hostname behind the request's IP address, then look that hostname back up and confirm it resolves to the same IP. A match confirms the request is real.

Faster in practice: match the request IP against the vendor's published IP list. OpenAI, Perplexity and Apple all publish machine-readable JSON files at their own domains for exactly this purpose; Anthropic publishes the equivalent at claude.com. Grep your access logs for the bot tokens in the table above, then confirm the real hits against the current IP list rather than trusting the name in the log line.

Once you trust the log line, the same data answers a more useful question than "did it visit": what is it doing here. Group verified hits by path and count them. A guide page ClaudeBot revisits weekly but OAI-SearchBot never touches tells you which system has the page in active rotation. No analytics dashboard shows that on its own. Cross-reference the same rows against the response code. A bot repeatedly hitting a URL that returns a 404, a 500 or a redirect chain is wasting its visits on a broken path, the AI-crawler version of the waste covered in crawl budget and log files.

Also watch the ratio between crawl volume and any referral traffic you can attribute to a given AI system, not just the raw hit count. Cloudflare's analysis of traffic across its network, published in August 2025, found Anthropic's crawlers had visited tens of thousands of pages for every visitor referred back to a site. It also noted that Anthropic's bots "still lag in verification" compared with other vendors. So spoofed traffic claiming to be ClaudeBot is easier to slip past a log check that only reads the user-agent string. That is a practical argument for the IP-based verification above.

A laptop screen showing lines of code, representing the server logs that prove whether an AI crawler visited

Bytespider and the crawlers with no published rulebook

Not every bot in the "Others" row of the table above plays by documented rules, and Bytespider, ByteDance's crawler, is the clearest example. Unlike OpenAI, Anthropic, Google, and Apple, ByteDance doesn't appear to have published an official crawler documentation page, a stated purpose, or a machine-readable IP range for Bytespider. Without a published IP list, there's no reliable way to confirm that a request identifying itself as Bytespider came from ByteDance rather than from anyone who copied the user-agent string into a scraper. The verification method that works for OpenAI, Anthropic and Perplexity (matching the request IP against the vendor's published list) isn't available here.

What is measurable is Bytespider's traffic volume, and it has swung sharply rather than following a steady trend. Cloudflare's own network data put Bytespider at 14.1% of combined AI and search crawler traffic in July 2024, then at 2.4% a year later in July 2025, a large single-year swing. That volatility, combined with the missing IP list, makes a Disallow rule in robots.txt a weaker guarantee for Bytespider than for a vendor with verifiable infrastructure. The rule only stops a crawler that identifies itself honestly and chooses to honour robots.txt. Site owners who want a firmer block usually add one at the server or CDN layer: a WAF rule matching the user-agent string, or blocking the IP ranges their hosting or CDN provider reports for ByteDance traffic.

The same caution extends to the rest of that row. Amazonbot, Meta-ExternalAgent, and CCBot each publish some level of documentation from their operators, but the safe default for any less-familiar bot is the same one that applies above: verify a claimed visit by IP against a real published list when one exists, and treat the name in a log line as a claim to check, not a fact to trust.

Access is the floor, not the strategy

Access and rendering are the floor, not the strategy. A page an AI crawler can't reach or can't read never gets a chance to be cited, no matter how well it's written. Fixing that is mechanical: server-rendered content, a robots.txt policy that distinguishes training from search, and a firewall that isn't silently blocking compliant bots. What happens after a page is readable, whether it gets cited, is a separate question, covered in how to show up in AI search results.

Sources: OpenAI's developer documentation at developers.openai.com/api/docs/bots, Anthropic's crawler support article at support.claude.com, and Perplexity's, Apple's, and Google's own current developer documentation for each crawler; Vercel & MERJ's independent analysis of AI-crawler traffic (December 2024); Cloudflare's blog post on crawl-to-referral ratios and crawler traffic share (August 2025); Cloudflare's reporting on Perplexity crawling behaviour.

New to the terms here? See crawling, robots.txt and GEO in the glossary. For the technical-SEO side of crawling and indexing, see crawling vs indexing.