Skip to content
Kuraib Ali

SEO guides

The technical basics that decide whether a page can be found, crawled, indexed and ranked, before content quality or AI search come into it.

By Kuraib AliPublished Updated

Search engine optimisation (SEO) is the practice of making a website easy for a search engine to find, understand and consider worth showing to the person who typed a query.

A page that can't be crawled doesn't get indexed. A page that isn't indexed can't rank, no matter how well it's written. That's the part most advice skips past, and it's the part that breaks sites most often.

What belongs under SEO

Crawling & indexing. Before a page can rank, a search engine's bot has to find it, read it, and decide it's worth storing. A broken internal link, a misconfigured robots.txt, or JavaScript hiding content from a crawler can all keep a page invisible, even when it looks fine to a human visitor.

Core Web Vitals. Google's specific measurements of real page experience: how fast the main content shows up, how quickly the page responds to a click, and how much it visually shifts while loading. They are a ranking factor.

On-page fundamentals. Titles, headings, and canonical tags aren't decoration. They're how a search engine, and increasingly an AI system, figures out what a page is about, and which version to treat as the real one when duplicates exist.

Technical SEO. The broader mechanics search engines depend on to render and understand a site correctly: rendering, structured data, site architecture: the plumbing underneath everything else here.

An architect's hand drafting a blueprint with a pencil and ruler, representing the planning technical SEO work requires

The order that matters

If a site has a real problem, it's almost always in this order: can it be crawled and indexed at all, is it fast enough not to lose people, and only then does on-page structure start to matter. Jumping straight to "on-page SEO" while a site has an indexing problem is a common, expensive mistake. New title tags don't help a page Google never sees.

White concrete stairs ascending step by step, representing the fixed order technical SEO fixes need to happen in: crawlability first, then speed, then on-page structure

How to diagnose a technical SEO problem

Most technical SEO work isn't creative, it's diagnostic: finding which of a small number of known failure points is responsible before touching anything. A repeatable order, using Google Search Console as the primary source of truth rather than guesswork:

  1. Check the Page indexing report first, not rankings. Search Console's Page indexing report shows why a URL isn't indexed. "Crawled, currently not indexed", "Discovered, currently not indexed", "Blocked by robots.txt", "Excluded by noindex tag" and "Duplicate without user-selected canonical" are the categories that explain most missing pages. Guessing at the cause before reading this report wastes time on the wrong fix.
  2. Confirm the page is reachable and renders correctly. Use the URL Inspection tool's "Live Test" to see the page exactly as Googlebot renders it, not as it looks in a browser. A page that depends on client-side JavaScript to inject its main content can pass a human eyeball check and still render empty to a crawler. See JavaScript SEO for what Googlebot does and doesn't execute.
  3. Rule out a canonicalisation conflict. If a page exists but a different URL is indexed instead, check for duplicate or near-duplicate content, an incorrect rel=canonical, or parameter variants competing with the clean URL. This is a distinct failure mode from not being indexed at all. See canonical tags.
  4. Check for accidental crawl blocks. A Disallow rule left over from staging, a noindex tag added during a redesign and never removed, or an AI-crawler rule that blocks Googlebot by mistake. See robots.txt and AI crawlers for exactly which directive controls which behaviour, since the names are easy to confuse.
  5. Only then look at Core Web Vitals and on-page structure. These affect how well an already-indexed page performs, not whether it gets indexed in the first place. Checking them before ruling out steps 1 to 4 is a common way to spend a week improving something that wasn't the actual problem.

A red Wrong Way road sign, representing the week of wasted effort that comes from fixing on-page details before ruling out an indexing or crawl problem

Common technical SEO mistakes

A handful of failure patterns account for most real technical SEO problems, and they tend to recur across otherwise very different sites:

  • Redirect chains that slow crawling. Each extra hop adds load time and makes it harder for Googlebot to reach the final URL. Sites that have gone through multiple redesigns or domain moves without cleaning up old redirects accumulate these silently. See 301 redirects and link equity for what the evidence shows about how much is lost, rather than the inflated numbers that circulate informally.
  • Crawl budget wasted on low-value URLs. Faceted navigation, internal search result pages, and near-duplicate parameter URLs can consume a disproportionate share of a large site's crawl budget, leaving new or updated pages crawled less often. Search Console's Crawl Stats report and your server logs show this directly. See crawl budget and log files.
  • A page that's indexed but never ranks for anything. Not every indexing problem is a crawl problem. A page can be fully indexed and still fail to rank because it doesn't sufficiently differentiate itself from a stronger, more established page already answering the same query. See indexed but not ranking for how to tell the difference.
  • Structured data that validates but doesn't help. Schema markup that's technically valid but uses a type Google no longer surfaces as a rich result is effort spent for no visible return. See schema types that still get rich results for what's current.

New to any of this? Start with crawling vs indexing. Nearly every other technical SEO problem downstream assumes that distinction is already clear.

Sixteen guides are published so far, in the order they go deeper into each piece:

Looking for the AI-search side of this: AEO, GEO, AI Overviews? See AI search.

Frequently asked questions

What's the difference between technical SEO and on-page SEO?

Technical SEO covers whether a search engine can crawl, render, and index a page at all: robots.txt, JavaScript rendering, canonical tags, site architecture. On-page SEO covers how well an already-indexed page communicates what it's about: titles, headings, content depth. A page can be perfectly optimised on-page and still never rank if a technical issue keeps it out of the index in the first place, which is why technical issues get fixed first.

Why is my page indexed but not ranking for anything?

Being indexed only means Google has stored a copy of the page; it says nothing about whether that page is considered the best answer for any given query. The two most common causes are content that doesn't sufficiently differentiate itself from an already-dominant competing page, and a canonicalisation conflict sending ranking signal to a different URL than the one meant to rank. See indexed but not ranking for the full diagnostic path.

Do Core Web Vitals affect rankings?

Google's own documentation states plainly that Core Web Vitals are used by its ranking systems, and separately states that a good score alone doesn't guarantee top rankings. In practice, they function as a real but secondary signal, most useful as a tiebreaker between pages that are already answering a query about equally well, not as a substitute for content that answers the query at all.

How long does it take for a technical SEO fix to show results?

It depends entirely on what was fixed. Removing a robots.txt block or a noindex tag can lead to recrawling within days once resubmitted in Search Console, but a full ranking response, if any, typically takes weeks as Google reassesses the page against competing results. Fixes to crawl budget or site-wide canonicalisation issues on larger sites can take longer simply because there's more to recrawl.

Can AI crawlers see my site if Googlebot can?

Not automatically. Googlebot, GPTBot, ClaudeBot, and PerplexityBot are separate crawlers with independently configurable robots.txt rules, and a site can allow one while blocking another without realising it. See robots.txt and AI crawlers for exactly which bot controls what, since names like Google-Extended are commonly misread as crawlers when they're permission flags on content Googlebot already fetched.