AI search guides
How AI systems decide what to read, trust and cite, covering answer engine optimisation (AEO), generative engine optimisation (GEO) and AI Overviews.
By Kuraib AliPublished Updated
- AEO, GEO and SEO: what each term means9 min read
- Do AI crawlers read your site?11 min read
- Do AI Overviews reduce clicks to websites?8 min read
- Does llms.txt work for AI search?10 min read
- Generative AI vs AI agents vs agentic AI: what separates them14 min read
- How ChatGPT decides what to cite9 min read
- How to show up in AI search results in 202613 min read
- How to track AI referral traffic from ChatGPT and other AI assistants in GA49 min read
- Is AI search optimisation different from SEO?9 min read
- What does Bing's AI Performance report show about Copilot citations?9 min read
- What is Web Bot Auth, and how do AI agents prove who they are?10 min read
AI search means a system reads across multiple sources and gives a synthesised answer, instead of returning a list of links for a person to click through. Google's AI Overviews, ChatGPT, and Perplexity each do a version of this. The mechanics differ, but the underlying question is the same for all of them: can this system access the page, understand what it says, and trust it enough to use it as a source.
That's a different bar than traditional ranking. A page can rank well in classic search and still never get pulled into an AI-generated answer, because AI systems apply their own filtering on top of, not instead of, normal SEO fundamentals.
What belongs under AI search
AEO. Answer Engine Optimisation. Structuring content so an AI system can lift out a direct, accurate answer without interpreting an entire page. Mostly a writing and structure problem: does the page state the answer clearly, near the top, in a form that can stand alone.
GEO. Generative Engine Optimisation. Making sure AI systems can physically access and read a page, and have a reason to trust it as a source. Closer to a technical-access problem: crawler permissions, rendering, and consistent, verifiable authorship.
AI Overviews & AI Mode. Google's specific AI-generated result formats. Both draw from multiple sources at once and decide, per query, whether to show an AI answer above or instead of a normal results list.
AI citations. What gets a page referenced inside an AI-generated answer: not just used silently in the background, but explicitly named or linked as a source.
Access comes before persuasion
A page an AI system can't read at all will never get cited, no matter how well-written the answer is inside it. Check GEO fundamentals (crawler access, rendering) before spending time optimising AEO-style answer structure.
How AI systems retrieve and cite content
The mechanism differs by product, but the general pipeline behind most AI-generated answers has three stages, and a failure at any one of them means the page never gets used, regardless of how good the content is:
- Crawl access. Each AI operator runs its own named crawler: OpenAI's
GPTBotandOAI-SearchBot, Anthropic'sClaudeBot, Perplexity'sPerplexityBot. These are independent from Googlebot and from each other; a site can allow Googlebot while unintentionally blocking every AI crawler, or vice versa.Google-ExtendedandApplebot-Extendedaren't crawlers at all. They're permission flags on content already fetched by Googlebot and Applebot.Google-Extendedcontrols whether content can be used for Gemini apps and Vertex AI. Google says it does not affect a site's inclusion in Google Search, so it does not control AI Overviews or AI Mode. See robots.txt and AI crawlers for the full breakdown, since this is the single most commonly misconfigured piece. - Rendering. A crawler that fetches a page's raw HTML still needs the actual answer-bearing content to be present in what it fetches. Content injected client-side after page load, with no server-rendered fallback, can be invisible to crawlers that don't execute JavaScript the same way a browser does. See JavaScript SEO for exactly what Googlebot renders and what several AI crawlers don't.
- Selection and citation. Once content is accessible and rendered, it competes with every other accessible source for inclusion in a generated answer. This is where AEO-style structure (a direct answer near the top, clear entity naming, content that stands alone as a citable unit) matters. It's also where signals related to E-E-A-T (experience, expertise, authoritativeness and trust), such as consistent authorship and verifiable claims, plausibly influence which of several accessible sources gets picked.
What is verified and what is still unsettled
A lot of AEO and GEO advice states things with more confidence than the evidence supports.
Verified, from the operators' own documentation: which crawlers exist and what each one is for (training vs. search-index vs. live user fetch); that Google-Extended and Applebot-Extended are permission flags, not crawlers; that AI Overviews and AI Mode draw from multiple sources and apply filtering on top of normal ranking, not instead of it; that Search Console now has a Generative AI performance report, separate from the standard Search performance report.
Reasoned inference, not directly confirmed by any operator: that clear, front-loaded answers and well-structured tables improve citation odds. This is a reasonable extrapolation from how retrieval systems work, and it fits what AEO practitioners report. But no operator has published a controlled study measuring it. Treat any exact citation-rate lift claimed for a particular technique with scepticism.
Unsettled: how much weight, if any, each AI system gives to traditional backlink authority versus content structure versus freshness when selecting sources for a specific answer. It varies by system, likely changes without announcement, and no outside party can measure it precisely. Treat any page that states a specific ranking weight for AI citations, mine included, as making a claim it can't back up.
Ten guides and one research note are published so far:
- AEO, GEO and SEO: what each term means
- Do AI crawlers read your site?
- How to show up in AI search results
- Generative AI vs AI agents vs agentic AI
- Does llms.txt work for AI search?
- Do AI Overviews reduce clicks?
- Tracking AI referral traffic in GA4
- How ChatGPT decides what to cite
- What Bing's AI Performance report shows about Copilot citations
- Web Bot Auth: how AI agents sign requests
- Research note: is AI search optimisation different from SEO?
Looking for the technical-SEO side of this: crawling, indexing, Core Web Vitals? See SEO.
Frequently asked questions
What's the difference between AEO and GEO?
AEO (Answer Engine Optimisation) is a writing and structure problem: does the page state a direct, accurate answer clearly enough that an AI system can lift it out without interpreting the whole page. GEO (Generative Engine Optimisation) is closer to a technical-access problem: can AI crawlers reach and render the page, and is there enough verifiable authorship for the system to trust it as a source. A page can be excellent on one and fail on the other.
If I block AI crawlers, does that affect my Google ranking?
Not directly. Google's ordinary ranking and indexing runs on Googlebot, which is separate from Google-Extended (the AI-training permission flag) and from other operators' crawlers like GPTBot and ClaudeBot. Blocking Google-Extended, GPTBot, or ClaudeBot doesn't touch standard Google Search rankings. It only affects whether that content can be used to train or ground those specific AI products. See robots.txt and AI crawlers for exactly which rule controls what.
Does having good SEO automatically mean good AI search visibility?
No, but it's a prerequisite, not a substitute. A page that's crawlable, well-structured, and already ranking well has cleared most of the access barrier AI systems apply. Beyond that, AI systems layer additional filtering on top of ordinary ranking signals, which is why a page can rank normally in Google and still never get pulled into an AI-generated answer.
Can I check whether AI crawlers are visiting my site?
Yes, the same way you'd check for any bot: server log analysis showing requests from user agents like GPTBot, ClaudeBot, or PerplexityBot. A crawler being allowed in robots.txt doesn't guarantee it's visiting. Log files are the only direct evidence of real crawl activity, the same principle covered in crawl budget and log files for traditional SEO.
Is AI Overviews the same thing as a featured snippet?
No. A featured snippet pulls one excerpt from a single source page. AI Overviews synthesise an answer across multiple sources at once and decide, per query, whether to show that synthesised answer above or instead of the normal results list. A page can be well-optimised for featured snippets and still not be one of the sources an AI Overview chooses to draw from.