Do AI Crawlers Actually Read Your Site?

Most of them, no. Not the way a browser does. The AI crawlers behind ChatGPT, Claude, and Perplexity split into three different jobs with three different sets of rules, and independent testing on real production traffic found that none of the major ones render JavaScript. If your content only appears after a script runs, it's invisible to almost every AI system that isn't Google.

Not one crawler: three jobs

"AI crawler" gets used as if it's one thing. It isn't. Every major AI company now runs multiple bots, split across three distinct jobs, and the job determines what blocking it actually does to you:

1. Training crawlers collect content to train future models. Blocking these opts you out of a model's underlying knowledge. It does nothing to your visibility today.
2. Search-index crawlers build the index a chatbot's search feature draws from. Blocking these is the one that actually costs you citations.
3. User-triggered fetchers run when a real person asks the assistant to open or check a specific page. Several of these don't reliably respect robots.txt at all, because the request is initiated by a person, not an automated crawl.

Treating all three as "the AI bots" and blanket-blocking them is the single most common mistake site owners make, and it usually blocks the wrong one.

Who's actually out there

Verified against each company's own current developer documentation.

Do they render JavaScript?

Mostly, no. Analysis of hundreds of millions of real AI-crawler requests (Vercel and MERJ, December 2024) found no evidence of JavaScript execution from GPTBot, OAI-SearchBot, ChatGPT-User, or ClaudeBot. They sometimes download a page's JS files (GPTBot in roughly 11% of requests, ClaudeBot in roughly 24%), but they read the file as text and never run it.

Google is the exception: Googlebot's crawl-render-index pipeline does execute JavaScript, which is why a client-rendered page can rank normally on Google while showing up as a near-blank page to every other AI crawler. If your headline, body copy, or key facts only appear after a script runs, server-side rendering, static generation, or prerendering is what actually fixes that, not a meta tag or a plugin setting.

The robots.txt mistake most sites make

Three patterns account for most of the "why isn't my site showing up in ChatGPT" cases:

• Blocking search bots by accident. A rule meant to stop AI training (disallowing GPTBot, ClaudeBot) gets copy-pasted broadly and catches OAI-SearchBot or Claude-SearchBot too, the ones that actually feed citations.
• A firewall blocking what robots.txt allows. WAF and rate-limiting rules (Cloudflare, and similar tools from other providers) can return 429 or 403 responses to a bot your robots.txt technically permits. The bot can't solve a CAPTCHA, so the block is silent and permanent from its side.
• Assuming a disallow stops everything. It doesn't reliably stop ChatGPT-User or Perplexity-User, since those fire on a real person's request rather than an automated crawl.

The fix isn't a blanket allow or a blanket block. It's deciding your policy per job. Allow the search/retrieval bots if AI visibility matters to you. Decide separately whether you want your content used for training.

How to test whether a specific AI crawler can actually reach a page

Reading robots.txt only shows what's supposed to happen. Testing shows what actually happens at the server or CDN, and the two disagree more often than site owners expect, usually because of a firewall rule nobody connected to AI crawlers in the first place.

1. Fetch the page with the crawler's real user-agent string. curl -A "<full UA string>" https://yoursite.com/page -I sends a request identifying as that bot and returns only the response headers. OpenAI's current developer docs publish the exact strings: GPTBot identifies as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot, with separate strings for OAI-SearchBot, ChatGPT-User, and OAI-AdsBot. Anthropic's ClaudeBot identifies as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +https://www.anthropic.com. A 200 response means nothing at the server layer is actively blocking that identity. A 403, a 429, or a redirect into a challenge page means something is, regardless of what robots.txt says.
2. Don't stop at robots.txt when a block shows up. A page can be fully allowed in robots.txt and still return a non-200 to the same user agent, because a WAF rule, a rate limiter, or a bot-management product is filtering the request before robots.txt is ever consulted. That's a firewall problem, not a robots.txt problem, and it needs fixing in the firewall.
3. Allow for propagation lag before concluding a fix didn't work. OpenAI's own documentation states it can take roughly 24 hours for a robots.txt change to reach its crawling systems. Retesting five minutes after an edit and assuming it failed is a common false negative.
4. Treat the curl test as proof of server behavior, not proof of who actually visited. Anyone can send a request with GPTBot's exact user-agent string; the curl test only confirms what your server does when it sees that string. Confirming the real bot showed up requires checking access logs against the vendor's published IP ranges, covered next: OpenAI publishes per-bot JSON files (openai.com/gptbot.json, openai.com/searchbot.json, and similarly named files for its other bots), and Anthropic publishes one combined list covering ClaudeBot, Claude-User, and Claude-SearchBot at claude.com/crawling/bots.json.

Run through those four steps per crawler that matters to you, rather than assuming that because one AI bot gets through, they all do. A rule scoped to GPTBot in a WAF product doesn't automatically catch ClaudeBot or PerplexityBot, and a security team that added AI-bot rules months ago may not have kept the list current as new bots launched.

How to check your own logs

A user-agent string alone proves nothing. It's a text field anyone can fake. Google's own documentation describes the reliable method, forward-confirmed reverse DNS: look up the hostname behind the request's IP address, then look that hostname back up and confirm it resolves to the same IP. A match confirms the request is genuine.

Faster in practice: match the request IP against the vendor's published IP list: OpenAI, Perplexity, and Apple all publish machine-readable JSON files at their own domains for exactly this purpose; Anthropic publishes the equivalent at claude.com. Grep your access logs for the bot tokens in the table above, then confirm the real hits against the current IP list rather than trusting the name in the log line.

Once you trust the log line, the same data answers a more useful question than "did it visit": what is it actually doing here. Group verified hits by path and count them. A guide page ClaudeBot revisits weekly but OAI-SearchBot never touches tells you concretely which system currently has the page in active rotation, information no analytics dashboard surfaces on its own. Cross-reference the same rows against the response code in that log line: a bot repeatedly hitting a URL that returns a 404, a 500, or a redirect chain is burning its own visits on a broken path, the AI-crawler version of the crawl-budget waste covered in crawl budget and log files. It's also worth watching the ratio between crawl volume and any referral traffic you can attribute back to a given AI system, not just the raw hit count. Cloudflare's own analysis of traffic across its network, published in August 2025, found Anthropic's crawlers had visited tens of thousands of pages for every single visitor referred back to a site, and separately noted that Anthropic's bots "still lag in verification" compared with other vendors, meaning spoofed traffic claiming to be ClaudeBot is comparatively easier to slip past a log check that only looks at the user-agent string. That's a direct, practical argument for the IP-based verification above rather than trusting the name in the log line at face value.

Bytespider and the crawlers with no published rulebook

Not every bot in the "Others" row of the table above plays by documented rules, and Bytespider, ByteDance's crawler, is the clearest example. Unlike OpenAI, Anthropic, Google, and Apple, ByteDance has not published an official crawler documentation page, a stated purpose, or a machine-readable IP range for Bytespider. That absence isn't just an inconvenience: without a published IP list, there's no reliable way to confirm that a request identifying itself as Bytespider actually came from ByteDance's own infrastructure rather than from anyone else who copied the same user-agent string into a scraper. The verification method that works for OpenAI, Anthropic, and Perplexity (matching the request IP against a vendor's own published list) simply isn't available for this one.

What is measurable is Bytespider's traffic volume, and it has swung sharply rather than following a steady trend. Cloudflare's own network data put Bytespider at 14.1% of AI-crawler traffic in July 2024, then at 2.4% a year later in July 2025, a large single-year swing the company's own reporting ties to broader shifts in how AI crawl traffic is distributed across vendors, not to any specific policy change on ByteDance's side. That volatility, combined with the missing IP list, is why blocking Bytespider through a Disallow rule in robots.txt alone is a weaker guarantee than it is for a vendor that publishes verifiable infrastructure: the rule only stops a crawler that identifies itself honestly and chooses to honor robots.txt in the first place. Site owners who want a firmer block typically add one at the server or CDN layer instead, a WAF rule matching the user-agent string or blocking whatever IP ranges their hosting or CDN provider reports for ByteDance traffic, rather than relying on the text file to do that work alone.

The same caution extends to the rest of that row. Amazonbot, Meta-ExternalAgent, and CCBot each publish some level of documentation from their operators, but the safe default for any less-familiar bot is the same one that applies throughout this guide: verify a claimed visit by IP against a real published list when one exists, and treat the name in a log line as a claim to check, not a fact to trust.

What this actually means

Access and rendering are the floor, not the strategy. A page an AI crawler can't reach or can't read never gets a chance to be cited, no matter how well it's written. Fixing that is mechanical: server-rendered content, a robots.txt policy that distinguishes training from search, and a firewall that isn't silently blocking compliant bots. What happens after a page is readable, whether it actually gets cited, is a separate question, covered in how to show up in AI search results.

Sources: OpenAI's developer documentation at developers.openai.com/api/docs/bots, Anthropic's crawler support article at support.claude.com, and Perplexity's, Apple's, and Google's own current developer documentation for each crawler; Vercel & MERJ's independent analysis of AI-crawler traffic (December 2024); Cloudflare's blog post on crawl-to-referral ratios and crawler traffic share (August 2025); Cloudflare's reporting on Perplexity crawling behavior. Full citations available on request via contact.

New to the terms here? See Crawling, Robots.txt, and GEO in the glossary. For the technical-SEO side of crawling and indexing, see crawling vs. indexing.