Crawl Budget and Log Files: What They Prove and What They Can't

Google's own documentation opens with an unusually direct disclaimer: crawl budget "is not something most publishers have to worry about." If new pages tend to get crawled the same day they're published, or the site has fewer than a few thousand URLs, it's genuinely not a relevant concern. For the sites where it does matter (large, frequently changing, or auto-generating URLs by the thousands), log files are the tool that replaces guessing with actual evidence of what's being crawled.

What is crawl budget, and do I need to worry about it?

Google defines crawl budget as the number of URLs Googlebot can and wants to crawl, combining a capacity limit (how much your server can handle) with a demand signal (how much Google actually wants to crawl, based on a page's popularity, uniqueness, and value). Google's own current guidance is specific about who this genuinely applies to, and the thresholds are smaller than a lot of general SEO advice implies:

• Large sites: 1 million or more unique pages, where content changes moderately often, roughly once a week.
• Medium or larger sites: 10,000 or more unique pages, where content changes very rapidly, daily or more.
• Any site, regardless of size, where Search Console classifies a large share of its total URLs as "Discovered - currently not indexed," since that status can point at a genuine crawl-scheduling bottleneck rather than a content problem.

If none of those describe the site in question, crawl budget isn't the bottleneck holding back indexing or rankings, no matter how often the term comes up in general SEO advice.

What actually increases crawl budget?

Google is direct about this too: "the only ways to increase your crawl budget are to increase your serving capacity for crawls, and… to increase the value of the content on your site to searchers." Several common tactics people try instead don't work, per Google's own "Myths and facts about crawling" page:

Common ways sites waste crawl budget

noindex as a budget-saving tactic: doesn't work. Google still requests the page, then drops it after seeing the noindex tag, wasting the crawl anyway.

Using robots.txt to "temporarily" reallocate budget: doesn't work either. Google won't shift budget elsewhere unless the site is already hitting its serving-capacity limit. robots.txt should be used for genuinely permanent exclusions, not short-term reallocation.

Soft 404s: a page that returns a 200 status but tells users "not found" keeps getting crawled indefinitely. A real 404 or 410 status is a strong, correct signal to stop.

What do log files actually prove?

Server logs record every request made to a site, including every real crawler hit: which pages, how often, and by which user agent. That makes them the only genuine ground truth for what's actually happening, as opposed to Search Console's sampled and delayed reporting, or general assumptions about crawler behavior. Log analysis reliably answers: which pages Googlebot or an AI crawler actually visits and how often, whether crawl frequency correlates with a page's real importance, and whether large sections of a site are being crawled rarely or not at all despite being linked and in the sitemap.

What log files can't answer on their own: why a page isn't ranking, or whether content quality is the problem. A page can be crawled constantly and still rank poorly. That's a relevance and quality question, not a crawl-frequency one. Log data shows access, not evaluation.

How do I actually read server logs?

Raw access logs (Apache, Nginx, or a CDN's equivalent, such as Cloudflare's logs) record one line per request: timestamp, requested path, status code, and user agent, at minimum. The practical steps:

1. Isolate real crawler traffic first. Filter to known bot user agents (Googlebot, GPTBot, ClaudeBot, PerplexityBot, and the rest), since the bulk of a raw log is ordinary visitor traffic that isn't relevant to this analysis.
2. Verify the requests are genuine, not spoofed. Anyone can set a user-agent string to say Googlebot. A real verification requires a reverse DNS lookup on the requesting IP confirming it resolves to a domain the operator actually owns (googlebot.com for Google, for instance), not just trusting the header.
3. Group by path and count frequency. This is what actually answers the crawl-budget question: which sections of the site get crawled often, which get crawled rarely, and which appear in the sitemap but never show up in the logs at all.
4. Cross-reference against the sitemap and site structure. A page that's linked, included in the sitemap, and still absent from months of crawler requests is the clearest sign of an access or crawl-priority problem worth investigating directly.

For sites too large to do this by hand, dedicated tools (Screaming Frog's Log File Analyser, JetOctopus, or a cloud log pipeline queried with something like BigQuery for very large sites) automate the filtering and aggregation, but the underlying method is the same four steps.

What the Crawl Stats report adds before you touch raw logs

Raw logs are the ground truth, but processing them, even with a dedicated tool, is still work. Search Console's Crawl Stats report gives a faster first read before pulling the raw data at all. It splits every recorded crawl request into four breakdowns: by response (the HTTP status code distribution, as a percentage of total crawl requests, where 200s should dominate on a healthy site), by file type (what share of crawl activity went to HTML documents versus images, CSS, JavaScript, and other resource types), by purpose (Discovery, meaning a URL Google is crawling for the first time, versus Refresh, meaning a recrawl of a page it already knows), and by Googlebot type (Smartphone, Desktop, Image, Video, Page resource load, AdsBot, StoreBot, and a general "other agent type" bucket).

None of that replaces log analysis. The numbers are aggregated site-wide and don't show individual paths, so they can't answer which specific section is being under-crawled. What the report is genuinely good for is spotting an obvious shape problem before spending time on raw data: a site where Refresh crawls dominate Discovery crawls for months after a large batch of new URLs went live, for instance, or a file-type breakdown showing an unexpectedly large share of crawl activity going to resource files instead of HTML documents. Either pattern is a reason to go pull the logs, not a substitute for pulling them.

How crawl budget interacts with JavaScript rendering cost

Crawling and rendering are two different steps, and that distinction matters directly for crawl budget on JavaScript-heavy sites. Google's own JavaScript SEO documentation describes a two-stage process: Googlebot first crawls and queues a page (any page returning a 200 status code gets queued for rendering, unless a robots meta tag says not to index it), and only once Google's rendering resources allow does a headless Chromium instance load the page and execute its JavaScript, before indexing can happen. That queue can hold a page for several seconds or considerably longer, depending on how much rendering capacity Google is allocating at the time.

Google has been direct about not tracking a literal cost-per-page figure internally. In an August 2024 LinkedIn post, Google's Martin Splitt put it plainly: "We don't keep track of 'how expensive was this page for us?'" and told site owners, "You don't need to worry about the fact that rendering is expensive, we got you covered." That's a statement about Google's own internal accounting, not a claim that rendering is free in practice from a site's perspective. A page that needs a full JavaScript execution to reveal its actual content still goes through an extra processing stage that a plain HTML page skips entirely. Given that documented two-stage process, the expected log pattern on a JavaScript-heavy site is an initial Googlebot fetch of the raw, unrendered document, followed at some later point by a second hit once the render queue catches up, assuming rendering happens at all for a lower-priority URL; this is a reasoned inference from Google's own stated mechanism, not a specific timing Google has published, and the actual delay isn't a fixed, published number. Google's own JavaScript SEO guidance still recommends server-side rendering or pre-rendering for exactly this reason, stating plainly that it's "still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript."

For a crawl-budget-constrained site specifically, the practical read is this: reducing how much a page's real content depends on client-side JavaScript execution doesn't just help human visitors, it removes an entire processing stage crawlers otherwise have to queue and wait for. That's one more concrete lever inside "increase the value of the content" and "increase serving capacity," the only two things Google's own guidance names as genuine ways to raise effective crawl budget. See JavaScript SEO, Googlebot rendering, and AI crawlers for the rendering mechanics in full.

A practical crawl-budget audit, step by step

For a site that actually meets one of the thresholds above, here's how the pieces already covered fit together into an actual audit, rather than isolated techniques used in a vacuum:

1. Start in Search Console's Crawl Stats report, by response. A healthy site shows the large majority of crawl requests returning 200. A meaningful share of 4xx or 5xx responses, or a recent spike in either, is worth chasing first, since every one of those requests is real crawl capacity spent on a URL returning nothing useful.
2. Check the by-purpose split. An established, stable site should show Refresh crawls dominating Discovery crawls. If Discovery stays unusually high long after launch, URL count is often growing faster than intended, sometimes from an unplanned source like faceted navigation or parameter combinations quietly generating new crawlable paths.
3. Pull the raw logs for a representative window. A week is usually enough on a large, consistently trafficked site. Isolate verified crawler hits using the filtering and reverse-DNS verification steps above; don't trust the user-agent string alone.
4. Group by path and cross-reference against the sitemap and site structure, exactly as described above, to find sections that are linked and submitted but rarely or never actually show up in the logs.
5. Weigh low-crawl-frequency sections against their actual business value. A rarely crawled section made up of genuinely low-value pages (thin archive pages, old filtered views) isn't a problem worth fixing. A rarely crawled section that should be driving real traffic is the actual finding an audit like this exists to surface.
6. Check whether the under-crawled pages depend on JavaScript rendering to show their real content, using the pattern described above: an early raw-document fetch followed by a much later render-stage hit, or no second hit at all. If so, the rendering cost is likely the root cause, not a crawl-priority problem on its own.
7. Fix the highest-value gap first, then re-pull the logs a few weeks later to confirm crawl frequency actually shifted. Crawl-budget problems don't resolve instantly. Google's own crawl-demand signal responds to sustained evidence that a page is worth fetching, not to a single one-off fix.

Related: do AI crawlers actually read your site, indexed but not ranking, JavaScript SEO, Googlebot rendering, and AI crawlers.