Crawling vs. Indexing: The Difference That Actually Matters

Crawling is Google finding and downloading a page's content. Indexing is Google deciding whether to store that content in its search database. A page can be crawled without being indexed, and even indexed without ranking well. Crawling happens first, indexing depends on it, and treating the two as one step instead of two is what leads to fixing the wrong problem.

What is crawling?

Crawling is Googlebot finding a page and downloading what's on it. Google has no master list of every page online. It builds one through URL discovery: following links from pages it already knows, reading sitemaps, and revisiting sites it's crawled before. A URL can be known to Google before it's ever visited. A page Googlebot can successfully reach and download is what's meant by crawlable: a prerequisite for indexing, not a guarantee of it.

During the crawl, Google also renders the page, running its JavaScript in a recent version of Chrome. Content that only appears after JavaScript runs is invisible to Google if that step fails.

Crawling can be blocked before any of this happens: a server error, a network problem, or a robots.txt rule telling Googlebot not to visit a URL at all.

What is indexing?

Indexing happens after a successful crawl. Google analyzes what it downloaded and decides whether the page belongs in its index, the database of pages eligible to appear in search results. Whether a page qualifies is its indexability: a mix of signals (noindex tags, canonical choices, content quality) rather than a single on/off switch.

This is also where duplicate handling happens. If several pages say roughly the same thing, Google groups them and picks one canonical version to represent the group; the others still exist, they're just not the one shown.

Indexing isn't automatic. Google's own documentation states plainly that not every crawled page ends up in the index.

Crawling vs. indexing, at a glance

Can Google crawl a page without indexing it?

Yes, and Search Console reports two statuses that look similar but mean different things.

Discovered – currently not indexed means Google knows the URL exists but hasn't crawled it. Google's own documentation gives a specific typical reason: Google wanted to crawl the URL, expected that doing so would overload the site, and rescheduled the crawl. That's why the last-crawl date is empty for these pages.

Crawled – currently not indexed means Google crawled the page but didn't index it. That's the entire official meaning. The label itself doesn't state a reason, and doesn't confirm the page was reviewed or judged in any particular way. It only says the page may or may not be indexed later, with no need to resubmit it for crawling.

Practitioners widely report thin or duplicate content as a contributing factor when this status shows up, a pattern observed across many sites, not something the status label itself claims. Treat it as a starting point for checking the page, not as a diagnosis.

Every Page Indexing report status, not just the two everyone mentions

Discovered and Crawled – currently not indexed get most of the attention, but Search Console's Page Indexing report tracks a much longer list. Sorting the full set by which stage actually stops the page, crawling or indexing, makes the report far easier to read at a glance:

The pattern worth internalizing: everything above "Crawled – currently not indexed" is a crawling-stage problem, meaning Google either never reached the page or the server actively stopped it. Everything below is an indexing-stage decision Google made after successfully reading the page's content. Fixing a crawling-stage status means fixing access (a server error, a robots.txt rule, an authorization wall). Fixing an indexing-stage status is almost always a content, canonicalization, or duplication problem, and no amount of crawling fixes touch it.

Does robots.txt keep a page out of Google's index?

Not reliably. Robots.txt controls crawling. It tells Googlebot which URLs it may visit. It was never built to remove pages from the index.

If other pages link to a blocked page, Google can sometimes still index the URL using information from those linking pages, without crawling the blocked page itself. Google describes this as a small chance, not a routine outcome, but it's real, which is why robots.txt alone doesn't reliably keep a page out of search results. When it happens, the result has no snippet, since Google never read the content.

The clean way to think about it: robots.txt controls whether Google can crawl a URL. noindex tells Google not to index a page it can already reach. Combining them carelessly backfires. If robots.txt blocks a page, Google never gets far enough to see a noindex tag on it. Google's own guidance says directly that blocking a page with robots.txt will prevent noindex from being seen at all.

Do XML sitemaps guarantee indexing?

No, and Google's own sitemap documentation is direct about it: "submitting a sitemap is merely a hint: it doesn't guarantee that Google will download the sitemap or use the sitemap for crawling URLs on the site." A sitemap's actual job is narrower than it's often assumed to be. It tells Google which URLs exist and, through the loc value, which version of a URL a site owner considers the canonical one (see sitemap priority, changefreq, and lastmod for what the rest of the sitemap protocol does and doesn't influence). That's a discovery aid and a canonicalization signal, not a crawl instruction and not an indexing instruction.

In practice, a sitemap can meaningfully speed up discovery, especially for a new page with weak internal linking, or on a large site where crawling through links alone would take a while to reach every URL. What it can't do is force Google to crawl a listed URL on any particular schedule, or force Google to index a page once crawled. Both of those still run through the same crawl-capacity, crawl-demand, and indexing-quality decisions described throughout this guide. A URL sitting in a submitted, valid sitemap and still showing "Discovered – currently not indexed" months later isn't a contradiction. The sitemap did its one job, telling Google the URL exists, while the actual crawl decision remains governed by everything else covered above.

What stops crawling or indexing

Crawling can be stopped by a robots.txt rule, a server error or timeout, a network issue, or a login requirement Googlebot can't get past.

Indexing can be stopped by a noindex tag (only effective once the page has actually been crawled), thin or duplicate content, being grouped under another page's canonical, or a rendering failure that hides the content Google needs to read.

Diagnosing indexing problems on a large site

The single-URL workflow described throughout this guide, check the Page Indexing report, then confirm with URL Inspection, works fine for one page. It doesn't scale to a site with tens of thousands of URLs showing indexing problems at once, which needs a different starting point:

1. Start with the Page Indexing report's status breakdown, not individual URLs. Which non-indexed status has the most URLs attached to it? A large "Duplicate without user-selected canonical" bucket points at a site-wide canonicalization gap (parameter-driven URLs, faceted navigation, or session IDs producing near-identical pages), not thousands of individual content problems.
2. Segment by URL pattern or template, not by counting pages one at a time. Large-site indexing problems tend to cluster by template (category pages, filtered listings, paginated archives, tag pages) rather than spreading evenly across the whole site. Identifying which template dominates a status shows where to fix the underlying logic once, instead of patching thousands of pages individually.
3. Check whether the affected template is genuinely link-reachable, not just present in the sitemap. A template linked only from a sitemap, with no real path to it through the site's actual navigation or content, is a weaker discovery signal than one with genuine internal links, and it shows up in exactly this kind of large-scale indexing gap.
4. Cross-reference against crawl logs for the same URL pattern. If a whole template rarely or never shows up in server logs (see crawl budget and log files), the indexing status is a downstream symptom. The actual problem sits upstream, at the crawling stage, not at the indexing decision the report is describing.
5. Prioritize by which pages would actually earn their place in the index, not by raw URL count. Thousands of thin, near-duplicate URLs showing "Crawled – currently not indexed" on a large site is often Google's quality judgment working as intended, not a bug to fix. The URLs worth chasing are the ones with real, differentiated content stuck in the same bucket.

What to actually check

Work through it in order. Is the URL known to Google at all? The Page Indexing report shows this. Has it been crawled? The same report, or URL Inspection for one page at a time, shows the last crawl date. Is it indexed? URL Inspection states this directly, with a specific reason attached when it isn't. From there: is robots.txt blocking the URL, is a noindex tag present, and if neither applies, is the content thin, duplicated, or dependent on JavaScript that didn't render.

Two things people still get wrong

"My page is indexed, so it should rank." Indexed means eligible to appear. Whether it actually shows up, and where, is a separate stage (serving) governed by relevance and quality signals that have nothing to do with whether the page made it into the index.

"More crawling means better rankings." How often Google recrawls a page depends partly on how frequently it expects that page's content to change: a signal to read, not a lever to pull directly. A page crawled daily but never indexed provides no visibility advantage over one crawled occasionally and indexed cleanly.

The practical takeaway

A page that isn't showing up usually gives you a specific, checkable starting point: known but not crawled, crawled but not indexed, or indexed but not chosen for a given result. Search Console can tell you which stage you're dealing with. Diagnosing the right stage is the first step; the fix follows from there.

Related: crawl budget and log files, sitemap priority, changefreq, and lastmod, indexed but not ranking.