Skip to content
Kuraib Ali

Crawl budget and log files: what they prove and what they can't

Server logs show exactly what a crawler did, not why. What log-file analysis can and can't tell you.

By Kuraib AliPublished Updated 10 min read

Close-up of server cooling fans in a vibrant data centre, representing the crawl capacity a site's server can handle
Photo: Unsplash

Google's 2017 post on crawl budget opened with a direct disclaimer: crawl budget "is not something most publishers have to worry about." Its current crawl budget guide makes the same point, telling most site owners "you don't need to read this guide." If new pages tend to get crawled the same day they're published, or the site has fewer than a few thousand URLs, it's not a relevant concern. For the sites where it does matter (large, frequently changing, or auto-generating URLs by the thousands), log files are the tool that replaces guessing with evidence of what's being crawled.

What is crawl budget, and do I need to worry about it?

Google defines crawl budget as the number of URLs Googlebot can and wants to crawl, combining a capacity limit (how much your server can handle) with a demand signal (how much Google wants to crawl, based on a page's popularity, uniqueness, and value). Google's own current guidance is specific about who this applies to, and the thresholds are smaller than a lot of general SEO advice implies:

  • Large sites: 1 million or more unique pages, where content changes moderately often, roughly once a week.
  • Medium or larger sites: 10,000 or more unique pages, where content changes very rapidly, daily or more.
  • Any site, regardless of size, where Search Console classifies a large share of its total URLs as "Discovered, currently not indexed", since that status can point at a real crawl-scheduling bottleneck rather than a content problem.

If none of those describe the site in question, crawl budget isn't the bottleneck holding back indexing or rankings, no matter how often the term comes up in general SEO advice.

What increases crawl budget?

Google is direct about this too. Its current guide says "There are two ways to increase crawl budget": add more server resources, and optimise the quality of your content. Several common tactics people try instead don't work, per Google's own "Myths about crawling" page:

ClaimGoogle's position
Compressing sitemaps increases crawl budgetFalse. The file still has to be fetched either way, no meaningful savings
Google prefers fresher content genericallyFalse. Content is rated by quality regardless of age; trivial edits to fake freshness add no value
Google dislikes URL parametersFalse. Google's own words: "We can crawl parameters."
Site speed affects crawl budgetTrue. Faster loading and rendering means more can be crawled in the same window

A cracked vintage wall clock, representing how crawl budget is a finite allowance a site can waste or spend well

Common ways sites waste crawl budget

noindex as a budget-saving tactic: doesn't work. Google still requests the page, then drops it after seeing the noindex tag, wasting the crawl anyway.

Using robots.txt to "temporarily" reallocate budget: doesn't work either. Google won't shift budget elsewhere unless the site is already hitting its serving-capacity limit. robots.txt should be used for permanent exclusions, not short-term reallocation.

Soft 404s: a page that returns a 200 status but tells users "not found" keeps getting crawled indefinitely. A real 404 or 410 status is a strong, correct signal to stop.

What do log files prove?

Server logs record every request made to a site, including every real crawler hit: which pages, how often, and by which user agent. That makes them the only ground truth for what's happening, as opposed to Search Console's sampled and delayed reporting, or general assumptions about crawler behaviour. Log analysis reliably answers: which pages Googlebot or an AI crawler visits and how often, whether crawl frequency correlates with a page's real importance, and whether large sections of a site are being crawled rarely or not at all despite being linked and in the sitemap.

What log files can't answer on their own: why a page isn't ranking, or whether content quality is the problem. A page can be crawled constantly and still rank poorly. That's a relevance and quality question, not a crawl-frequency one. Log data shows access, not evaluation.

Vertical columns of glowing green digital code, representing the raw server log data crawl budget analysis is built from

How do I read server logs?

Raw access logs (Apache, Nginx, or the equivalent from a content delivery network (CDN) such as Cloudflare) record one line per request: timestamp, requested path, status code, and user agent, at minimum. The practical steps:

  1. Isolate real crawler traffic first. Filter to known bot user agents (Googlebot, GPTBot, ClaudeBot, PerplexityBot, and the rest), since the bulk of a raw log is ordinary visitor traffic that isn't relevant to this analysis.
  2. Verify the requests are real, not spoofed. Anyone can set a user-agent string to say Googlebot. Proper verification requires a reverse DNS (domain name system) lookup on the requesting IP confirming it resolves to a domain the operator owns (googlebot.com for Google, for instance), not just trusting the header.
  3. Group by path and count frequency. This is what answers the crawl-budget question: which sections of the site get crawled often, which get crawled rarely, and which appear in the sitemap but never show up in the logs at all.
  4. Cross-reference against the sitemap and site structure. A page that's linked, included in the sitemap, and still absent from months of crawler requests is the clearest sign of an access or crawl-priority problem worth investigating directly.

For sites too large to do this by hand, dedicated tools (Screaming Frog's Log File Analyser, JetOctopus, or a cloud log pipeline queried with something like BigQuery for very large sites) automate the filtering and aggregation, but the underlying method is the same four steps.

What the Crawl Stats report adds before you touch raw logs

Raw logs are the ground truth, but processing them, even with a dedicated tool, is still work. Search Console's Crawl Stats report gives a faster first read before pulling the raw data at all. It splits every recorded crawl request into four breakdowns:

  • By response: the HTTP status code distribution, as a percentage of total crawl requests. 200s should dominate on a healthy site.
  • By file type: what share of crawl activity went to HTML documents versus images, CSS, JavaScript and other resource types.
  • By purpose: Discovery, meaning a URL Google is crawling for the first time, versus Refresh, meaning a recrawl of a page it already knows.
  • By Googlebot type: Smartphone, Desktop, Image, Video, Page resource load, AdsBot, StoreBot, and a general "other agent type" bucket.

None of that replaces log analysis. The numbers are aggregated site-wide and don't show individual paths, so they can't answer which specific section is being under-crawled. What the report is good for is spotting an obvious shape problem before spending time on raw data: a site where Refresh crawls dominate Discovery crawls for months after a large batch of new URLs went live, for instance, or a file-type breakdown showing an unexpectedly large share of crawl activity going to resource files instead of HTML documents. Either pattern is a reason to go pull the logs, not a substitute for pulling them.

How crawl budget interacts with JavaScript rendering cost

Crawling and rendering are two different steps, and that distinction matters directly for crawl budget on JavaScript-heavy sites. Google's own JavaScript SEO documentation describes a two-stage process: Googlebot first crawls and queues a page (any page returning a 200 status code gets queued for rendering, unless a robots meta tag says not to index it), and only once Google's rendering resources allow does a headless Chromium instance load the page and execute its JavaScript, before indexing can happen. That queue can hold a page for several seconds or considerably longer, depending on how much rendering capacity Google is allocating at the time.

However Google accounts for rendering internally, rendering isn't free in practice from a site's perspective. A page that needs a full JavaScript execution to reveal its content still goes through an extra processing stage that a plain HTML page skips. Given that two-stage process, the expected log pattern on a JavaScript-heavy site is an initial Googlebot fetch of the raw, unrendered document. A second hit follows later once the render queue catches up, if rendering happens at all for a lower-priority URL. This is my inference from Google's stated mechanism. Google hasn't published a timing, and the delay isn't fixed. Google's own JavaScript SEO guidance still recommends server-side rendering or pre-rendering for exactly this reason, stating plainly that it's "still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript."

For a crawl-budget-constrained site specifically, the practical read is this: reducing how much a page's real content depends on client-side JavaScript execution doesn't just help human visitors, it removes an entire processing stage crawlers otherwise have to queue and wait for. That's one more concrete lever inside "increase the value of the content" and "increase serving capacity," the only two ways Google's own guidance names to raise effective crawl budget. See JavaScript SEO, Googlebot rendering, and AI crawlers for the rendering mechanics in full.

A hand holding a pen over a blue clipboard, representing the step-by-step checklist a crawl-budget audit works through

A practical crawl-budget audit, step by step

For a site that meets one of the thresholds above, here's how the pieces already covered fit together into an audit:

  1. Start in Search Console's Crawl Stats report, by response. A healthy site shows the large majority of crawl requests returning 200. A meaningful share of 4xx or 5xx responses, or a recent spike in either, is worth chasing first, since every one of those requests is real crawl capacity spent on a URL returning nothing useful.
  2. Check the by-purpose split. An established, stable site should show Refresh crawls dominating Discovery crawls. If Discovery stays unusually high long after launch, URL count is often growing faster than intended, sometimes from an unplanned source like faceted navigation or parameter combinations quietly generating new crawlable paths.
  3. Pull the raw logs for a representative window. A week is usually enough on a large, consistently trafficked site. Isolate verified crawler hits using the filtering and reverse-DNS verification steps above; don't trust the user-agent string alone.
  4. Group by path and cross-reference against the sitemap and site structure, exactly as described above, to find sections that are linked and submitted but rarely or never show up in the logs.
  5. Weigh low-crawl-frequency sections against their business value. A rarely crawled section made up of low-value pages (thin archive pages, old filtered views) isn't a problem worth fixing. A rarely crawled section that should be driving real traffic is the finding an audit like this exists to surface.
  6. Check whether the under-crawled pages depend on JavaScript rendering to show their real content, using the pattern described above: an early raw-document fetch followed by a much later render-stage hit, or no second hit at all. If so, the rendering cost is likely the root cause, not a crawl-priority problem on its own.
  7. Fix the highest-value gap first, then re-pull the logs a few weeks later to confirm crawl frequency shifted. Crawl-budget problems don't resolve instantly. Google's own crawl-demand signal responds to sustained evidence that a page is worth fetching, not to a single one-off fix.

Related: do AI crawlers read your site, indexed but not ranking, JavaScript SEO, Googlebot rendering, and AI crawlers.