Crawling vs indexing: the difference explained
Crawling and indexing are separate steps, and mixing them up leads to the wrong fix. What each one means, and what Search Console statuses tell you.
By Kuraib AliPublished Updated 11 min read
Crawling is Google finding and downloading a page's content. Indexing is Google deciding whether to store that content in its search database. A page can be crawled without being indexed, and even indexed without ranking well. Crawling happens first, indexing depends on it, and treating the two as one step instead of two is what leads to fixing the wrong problem.
What is crawling?
Crawling is Googlebot finding a page and downloading what's on it. Google has no master list of every page online. It builds one through URL discovery: following links from pages it already knows, reading sitemaps, and revisiting sites it's crawled before. A URL can be known to Google before it's ever visited. A page Googlebot can successfully reach and download is what's meant by crawlable: a prerequisite for indexing, not a guarantee of it.
During the crawl, Google also renders the page, running its JavaScript in a recent version of Chrome. Content that only appears after JavaScript runs is invisible to Google if that step fails.
Crawling can be blocked before any of this happens: a server error, a network problem, or a robots.txt rule telling Googlebot not to visit a URL at all.
What is indexing?
Indexing happens after a successful crawl. Google analyses what it downloaded and decides whether the page belongs in its index, the database of pages eligible to appear in search results. Whether a page qualifies is its indexability: a mix of signals (noindex tags, canonical choices, content quality) rather than a single on/off switch.
This is also where duplicate handling happens. If several pages say roughly the same thing, Google groups them and picks one canonical version to represent the group; the others still exist, they're just not the one shown.
Indexing isn't automatic. Google's own documentation states plainly that not every crawled page ends up in the index.
Crawling vs indexing at a glance
| Crawling | Indexing | |
|---|---|---|
| What it is | Downloading a page's content | Analysing and storing that content |
| Comes first? | Yes | No, after a successful crawl |
| Can be skipped by | robots.txt rules, server/network errors | noindex, low content quality, rendering problems |
| Where you check it | Crawl Stats, URL Inspection | Page indexing report, URL Inspection |
| Guarantees the next step? | No | No, indexed doesn't mean ranked |
Can Google crawl a page without indexing it?
Yes, and Search Console reports two statuses that look similar but mean different things.
"Discovered, currently not indexed" means Google knows the URL exists but hasn't crawled it. Google's own documentation gives a specific typical reason: Google wanted to crawl the URL, expected that doing so would overload the site, and rescheduled the crawl. That's why the last-crawl date is empty for these pages.
"Crawled, currently not indexed" means Google crawled the page but didn't index it. The label itself doesn't state a reason, and doesn't confirm the page was reviewed or judged in any particular way. Google's current help text adds context rather than a diagnosis: "Google doesn't index all pages across the web, so this is a common message for many sites and doesn't necessarily indicate an issue."
Practitioners widely report thin or duplicate content as a contributing factor when this status shows up, a pattern observed across many sites, not something the status label itself claims. Treat it as a starting point for checking the page, not as a diagnosis.
The main Page indexing report statuses, not just the two everyone mentions
"Discovered, currently not indexed" and "Crawled, currently not indexed" get most of the attention, but Search Console's Page indexing report tracks a much longer list. Sorting the main ones by which stage stops the page, crawling or indexing, makes the report far easier to read at a glance:
| Status | Stage it belongs to | What it means |
|---|---|---|
| Server error (5xx) | Crawling | The server returned a 500-level error when Google requested the page |
| Redirect error | Crawling | A redirect chain too long, a redirect loop, or an otherwise broken redirect path |
| Blocked by robots.txt | Crawling | robots.txt disallowed the request; Google never fetched the page |
| Indexed, though blocked by robots.txt | Crawling | robots.txt blocked the request, but Google indexed the URL anyway from links pointing to it, without reading the page |
| Blocked due to unauthorised request (401) | Crawling | Googlebot hit an authorisation wall it couldn't get past |
| Blocked due to access forbidden (403) | Crawling | A 403 means credentials were provided but access was refused. Googlebot never sends credentials, so the server is returning this response incorrectly |
| Not found (404) | Crawling | The page returned a real 404 when requested |
| URL blocked due to other 4xx issue | Crawling | The server returned a 4xx error that isn't covered by another status in this list |
| Discovered, currently not indexed | Crawling | Google knows the URL exists but hasn't crawled it yet, typically a scheduling decision |
| Crawled, currently not indexed | Indexing | Google fetched the page and chose not to index it |
| URL marked "noindex" | Indexing | A noindex directive was present once Google read the page |
| Soft 404 | Indexing | The page returns a normal 200 status, but its content reads as a "not found" message |
| Duplicate without user-selected canonical | Indexing | The page duplicates another, and no canonical was specified for the set |
| Duplicate, Google chose different canonical than user | Indexing | A canonical was specified, but Google indexed a different URL from the set instead |
| Alternate page with proper canonical tag | Indexing | Correctly marked as an alternate (an AMP or mobile version, for instance) of an already-indexed page |
| Page with redirect | Indexing | A non-canonical URL that redirects elsewhere |
| Page is indexed | The outcome, not a problem | The page cleared both stages and is in Google's index |
The pattern to remember: everything above "Crawled, currently not indexed" is a crawling-stage problem, meaning Google either never reached the page or the server actively stopped it. Everything below is an indexing-stage decision Google made after successfully reading the page's content. Fixing a crawling-stage status means fixing access (a server error, a robots.txt rule, an authorisation wall). Fixing an indexing-stage status is almost always a content, canonicalisation, or duplication problem, and no amount of crawling fixes touch it.
Does robots.txt keep a page out of Google's index?
Not reliably. Robots.txt controls crawling. It tells Googlebot which URLs it may visit. It was never built to remove pages from the index.
If other pages link to a blocked page, Google can sometimes still index the URL using information from those linking pages, without crawling the blocked page itself. Search Console reports it as "Indexed, though blocked by robots.txt", and it's why robots.txt alone doesn't reliably keep a page out of search results. When it happens, the result has no snippet, since Google never read the content.
The clean way to think about it: robots.txt controls whether Google can crawl a URL. noindex tells Google not to index a page it can already reach. Combining them carelessly backfires. If robots.txt blocks a page, Google never gets far enough to see a noindex tag on it. Google's own guidance says directly that blocking a page with robots.txt will prevent noindex from being seen at all.
Do XML sitemaps guarantee indexing?
No, and Google's own sitemap documentation is direct about it: "submitting a sitemap is merely a hint: it doesn't guarantee that Google will download the sitemap or use the sitemap for crawling URLs on the site." A sitemap's job is narrower than it's often assumed to be. It tells Google which URLs exist and, through the loc value, which version of a URL a site owner considers the canonical one (see sitemap priority, changefreq, and lastmod for what the rest of the sitemap protocol does and doesn't influence). That's a discovery aid and a canonicalisation signal, not a crawl instruction and not an indexing instruction.
In practice, a sitemap can meaningfully speed up discovery, especially for a new page with weak internal linking, or on a large site where crawling through links alone would take a while to reach every URL. What it can't do is force Google to crawl a listed URL on any particular schedule, or force Google to index a page once crawled. Both of those still run through the same crawl-capacity, crawl-demand, and indexing-quality decisions described above. A URL sitting in a submitted, valid sitemap and still showing "Discovered, currently not indexed" months later isn't a contradiction. The sitemap did its one job, telling Google the URL exists, while the crawl decision remains governed by everything else covered above.
What stops crawling or indexing
Crawling can be stopped by a robots.txt rule, a server error or timeout, a network issue, or a login requirement Googlebot can't get past.
Indexing can be stopped by a noindex tag (only effective once the page has been crawled), thin or duplicate content, being grouped under another page's canonical, or a rendering failure that hides the content Google needs to read.
Diagnosing indexing problems on a large site
The single-URL workflow above (check the Page indexing report, then confirm with URL Inspection) works for one page. It doesn't scale to a site with tens of thousands of URLs showing indexing problems at once, which needs a different starting point:
- Start with the Page indexing report's status breakdown, not individual URLs. Which non-indexed status has the most URLs attached to it? A large "Duplicate without user-selected canonical" bucket points at a site-wide canonicalisation gap (parameter-driven URLs, faceted navigation, or session IDs producing near-identical pages), not thousands of individual content problems.
- Segment by URL pattern or template, not by counting pages one at a time. Large-site indexing problems tend to cluster by template (category pages, filtered listings, paginated archives, tag pages) rather than spreading evenly across the whole site. Identifying which template dominates a status shows where to fix the underlying logic once, instead of patching thousands of pages individually.
- Check whether the affected template is link-reachable, not just present in the sitemap. A template linked only from a sitemap, with no real path to it through the site's actual navigation or content, is a weaker discovery signal than one with real internal links, and it shows up in exactly this kind of large-scale indexing gap.
- Cross-reference against crawl logs for the same URL pattern. If a whole template rarely or never shows up in server logs (see crawl budget and log files), the indexing status is a downstream symptom. The problem sits upstream, at the crawling stage, not at the indexing decision the report is describing.
- Prioritise by which pages would earn their place in the index, not by raw URL count. Thousands of thin, near-duplicate URLs showing "Crawled, currently not indexed" on a large site is often Google's quality judgement working as intended, not a bug to fix. The URLs worth chasing are the ones with real, differentiated content stuck in the same bucket.
Which stage to check, in order
Work through it in order. Is the URL known to Google at all? The Page indexing report shows this. Has it been crawled? The same report, or URL Inspection for one page at a time, shows the last crawl date. Is it indexed? URL Inspection states this directly, with a specific reason attached when it isn't. From there: is robots.txt blocking the URL, is a noindex tag present, and if neither applies, is the content thin, duplicated, or dependent on JavaScript that didn't render.
Two things people still get wrong
"My page is indexed, so it should rank." Indexed means eligible to appear. Whether it shows up, and where, is a separate stage (serving) governed by relevance and quality signals that have nothing to do with whether the page made it into the index.
"More crawling means better rankings." How often Google recrawls a page depends partly on how frequently it expects that page's content to change: a signal to read, not a lever to pull directly. A page crawled daily but never indexed provides no visibility advantage over one crawled occasionally and indexed cleanly.
Related: crawl budget and log files, sitemap priority, changefreq, and lastmod, indexed but not ranking.