Skip to content
Kuraib Ali

Crawling vs indexing: the difference explained

Crawling and indexing are separate steps, and mixing them up leads to the wrong fix. What each one means, and what Search Console statuses tell you.

By Kuraib AliPublished Updated 11 min read

A black and white network of connected dots, representing how Google discovers new pages by following links
Photo: Unsplash

Crawling is Google finding and downloading a page's content. Indexing is Google deciding whether to store that content in its search database. A page can be crawled without being indexed, and even indexed without ranking well. Crawling happens first, indexing depends on it, and treating the two as one step instead of two is what leads to fixing the wrong problem.

What is crawling?

Crawling is Googlebot finding a page and downloading what's on it. Google has no master list of every page online. It builds one through URL discovery: following links from pages it already knows, reading sitemaps, and revisiting sites it's crawled before. A URL can be known to Google before it's ever visited. A page Googlebot can successfully reach and download is what's meant by crawlable: a prerequisite for indexing, not a guarantee of it.

During the crawl, Google also renders the page, running its JavaScript in a recent version of Chrome. Content that only appears after JavaScript runs is invisible to Google if that step fails.

Crawling can be blocked before any of this happens: a server error, a network problem, or a robots.txt rule telling Googlebot not to visit a URL at all.

What is indexing?

Indexing happens after a successful crawl. Google analyses what it downloaded and decides whether the page belongs in its index, the database of pages eligible to appear in search results. Whether a page qualifies is its indexability: a mix of signals (noindex tags, canonical choices, content quality) rather than a single on/off switch.

This is also where duplicate handling happens. If several pages say roughly the same thing, Google groups them and picks one canonical version to represent the group; the others still exist, they're just not the one shown.

Indexing isn't automatic. Google's own documentation states plainly that not every crawled page ends up in the index.

Crawling vs indexing at a glance

CrawlingIndexing
What it isDownloading a page's contentAnalysing and storing that content
Comes first?YesNo, after a successful crawl
Can be skipped byrobots.txt rules, server/network errorsnoindex, low content quality, rendering problems
Where you check itCrawl Stats, URL InspectionPage indexing report, URL Inspection
Guarantees the next step?NoNo, indexed doesn't mean ranked

A hand pulling a card from a library's card catalogue, representing indexing: organising what's already been found

Can Google crawl a page without indexing it?

Yes, and Search Console reports two statuses that look similar but mean different things.

"Discovered, currently not indexed" means Google knows the URL exists but hasn't crawled it. Google's own documentation gives a specific typical reason: Google wanted to crawl the URL, expected that doing so would overload the site, and rescheduled the crawl. That's why the last-crawl date is empty for these pages.

"Crawled, currently not indexed" means Google crawled the page but didn't index it. The label itself doesn't state a reason, and doesn't confirm the page was reviewed or judged in any particular way. Google's current help text adds context rather than a diagnosis: "Google doesn't index all pages across the web, so this is a common message for many sites and doesn't necessarily indicate an issue."

Practitioners widely report thin or duplicate content as a contributing factor when this status shows up, a pattern observed across many sites, not something the status label itself claims. Treat it as a starting point for checking the page, not as a diagnosis.

The main Page indexing report statuses, not just the two everyone mentions

"Discovered, currently not indexed" and "Crawled, currently not indexed" get most of the attention, but Search Console's Page indexing report tracks a much longer list. Sorting the main ones by which stage stops the page, crawling or indexing, makes the report far easier to read at a glance:

StatusStage it belongs toWhat it means
Server error (5xx)CrawlingThe server returned a 500-level error when Google requested the page
Redirect errorCrawlingA redirect chain too long, a redirect loop, or an otherwise broken redirect path
Blocked by robots.txtCrawlingrobots.txt disallowed the request; Google never fetched the page
Indexed, though blocked by robots.txtCrawlingrobots.txt blocked the request, but Google indexed the URL anyway from links pointing to it, without reading the page
Blocked due to unauthorised request (401)CrawlingGooglebot hit an authorisation wall it couldn't get past
Blocked due to access forbidden (403)CrawlingA 403 means credentials were provided but access was refused. Googlebot never sends credentials, so the server is returning this response incorrectly
Not found (404)CrawlingThe page returned a real 404 when requested
URL blocked due to other 4xx issueCrawlingThe server returned a 4xx error that isn't covered by another status in this list
Discovered, currently not indexedCrawlingGoogle knows the URL exists but hasn't crawled it yet, typically a scheduling decision
Crawled, currently not indexedIndexingGoogle fetched the page and chose not to index it
URL marked "noindex"IndexingA noindex directive was present once Google read the page
Soft 404IndexingThe page returns a normal 200 status, but its content reads as a "not found" message
Duplicate without user-selected canonicalIndexingThe page duplicates another, and no canonical was specified for the set
Duplicate, Google chose different canonical than userIndexingA canonical was specified, but Google indexed a different URL from the set instead
Alternate page with proper canonical tagIndexingCorrectly marked as an alternate (an AMP or mobile version, for instance) of an already-indexed page
Page with redirectIndexingA non-canonical URL that redirects elsewhere
Page is indexedThe outcome, not a problemThe page cleared both stages and is in Google's index

The pattern to remember: everything above "Crawled, currently not indexed" is a crawling-stage problem, meaning Google either never reached the page or the server actively stopped it. Everything below is an indexing-stage decision Google made after successfully reading the page's content. Fixing a crawling-stage status means fixing access (a server error, a robots.txt rule, an authorisation wall). Fixing an indexing-stage status is almost always a content, canonicalisation, or duplication problem, and no amount of crawling fixes touch it.

Does robots.txt keep a page out of Google's index?

Not reliably. Robots.txt controls crawling. It tells Googlebot which URLs it may visit. It was never built to remove pages from the index.

If other pages link to a blocked page, Google can sometimes still index the URL using information from those linking pages, without crawling the blocked page itself. Search Console reports it as "Indexed, though blocked by robots.txt", and it's why robots.txt alone doesn't reliably keep a page out of search results. When it happens, the result has no snippet, since Google never read the content.

The clean way to think about it: robots.txt controls whether Google can crawl a URL. noindex tells Google not to index a page it can already reach. Combining them carelessly backfires. If robots.txt blocks a page, Google never gets far enough to see a noindex tag on it. Google's own guidance says directly that blocking a page with robots.txt will prevent noindex from being seen at all.

Do XML sitemaps guarantee indexing?

No, and Google's own sitemap documentation is direct about it: "submitting a sitemap is merely a hint: it doesn't guarantee that Google will download the sitemap or use the sitemap for crawling URLs on the site." A sitemap's job is narrower than it's often assumed to be. It tells Google which URLs exist and, through the loc value, which version of a URL a site owner considers the canonical one (see sitemap priority, changefreq, and lastmod for what the rest of the sitemap protocol does and doesn't influence). That's a discovery aid and a canonicalisation signal, not a crawl instruction and not an indexing instruction.

In practice, a sitemap can meaningfully speed up discovery, especially for a new page with weak internal linking, or on a large site where crawling through links alone would take a while to reach every URL. What it can't do is force Google to crawl a listed URL on any particular schedule, or force Google to index a page once crawled. Both of those still run through the same crawl-capacity, crawl-demand, and indexing-quality decisions described above. A URL sitting in a submitted, valid sitemap and still showing "Discovered, currently not indexed" months later isn't a contradiction. The sitemap did its one job, telling Google the URL exists, while the crawl decision remains governed by everything else covered above.

What stops crawling or indexing

Crawling can be stopped by a robots.txt rule, a server error or timeout, a network issue, or a login requirement Googlebot can't get past.

Indexing can be stopped by a noindex tag (only effective once the page has been crawled), thin or duplicate content, being grouped under another page's canonical, or a rendering failure that hides the content Google needs to read.

A close-up photograph of a spider, representing Googlebot's crawling process of discovering pages by following links

Diagnosing indexing problems on a large site

The single-URL workflow above (check the Page indexing report, then confirm with URL Inspection) works for one page. It doesn't scale to a site with tens of thousands of URLs showing indexing problems at once, which needs a different starting point:

  1. Start with the Page indexing report's status breakdown, not individual URLs. Which non-indexed status has the most URLs attached to it? A large "Duplicate without user-selected canonical" bucket points at a site-wide canonicalisation gap (parameter-driven URLs, faceted navigation, or session IDs producing near-identical pages), not thousands of individual content problems.
  2. Segment by URL pattern or template, not by counting pages one at a time. Large-site indexing problems tend to cluster by template (category pages, filtered listings, paginated archives, tag pages) rather than spreading evenly across the whole site. Identifying which template dominates a status shows where to fix the underlying logic once, instead of patching thousands of pages individually.
  3. Check whether the affected template is link-reachable, not just present in the sitemap. A template linked only from a sitemap, with no real path to it through the site's actual navigation or content, is a weaker discovery signal than one with real internal links, and it shows up in exactly this kind of large-scale indexing gap.
  4. Cross-reference against crawl logs for the same URL pattern. If a whole template rarely or never shows up in server logs (see crawl budget and log files), the indexing status is a downstream symptom. The problem sits upstream, at the crawling stage, not at the indexing decision the report is describing.
  5. Prioritise by which pages would earn their place in the index, not by raw URL count. Thousands of thin, near-duplicate URLs showing "Crawled, currently not indexed" on a large site is often Google's quality judgement working as intended, not a bug to fix. The URLs worth chasing are the ones with real, differentiated content stuck in the same bucket.

A large warehouse filled with rows of shelving, representing the scale a single-URL indexing check doesn't cover on a large site

Which stage to check, in order

Work through it in order. Is the URL known to Google at all? The Page indexing report shows this. Has it been crawled? The same report, or URL Inspection for one page at a time, shows the last crawl date. Is it indexed? URL Inspection states this directly, with a specific reason attached when it isn't. From there: is robots.txt blocking the URL, is a noindex tag present, and if neither applies, is the content thin, duplicated, or dependent on JavaScript that didn't render.

Two things people still get wrong

"My page is indexed, so it should rank." Indexed means eligible to appear. Whether it shows up, and where, is a separate stage (serving) governed by relevance and quality signals that have nothing to do with whether the page made it into the index.

"More crawling means better rankings." How often Google recrawls a page depends partly on how frequently it expects that page's content to change: a signal to read, not a lever to pull directly. A page crawled daily but never indexed provides no visibility advantage over one crawled occasionally and indexed cleanly.

Related: crawl budget and log files, sitemap priority, changefreq, and lastmod, indexed but not ranking.