Skip to content
Kuraib Ali

Does llms.txt work for AI search?

Google says llms.txt won't move AI Overviews or AI Mode. No major AI crawler has confirmed using it as a ranking signal either. Here's what the file does.

By Kuraib AliPublished Updated 10 min read

A minimal 3D text file icon, representing llms.txt itself: a plain markdown file sitting at a site's root
Photo: Unsplash

llms.txt is a markdown file, served at a site's root URL, that proposes a curated summary of a site's content plus links to markdown versions of key pages, meant to help a language model orient itself without parsing full HTML. Google's official guidance (published May 2026, with an llms.txt note added in June 2026) says plainly that "Google Search itself doesn't use them." So it isn't required or specially treated for AI Overviews or AI Mode. No major AI operator (OpenAI, Anthropic, Google, Meta) has publicly confirmed using it as a ranking or retrieval signal in production. It's a useful convenience in a narrower case: AI coding assistants reading developer documentation directly.

What is llms.txt?

llms.txt is a proposed markdown file, served at the root of a site, that gives a language model a curated summary of the site and links to markdown versions of its key pages. It is a community convention started by Jeremy Howard in September 2024, not a standard from a standards body. Google's guidance says Google Search itself does not use it.

Where did llms.txt come from?

Jeremy Howard, co-founder of Answer.AI and fast.ai, published the original proposal on 3 September 2024, on Answer.AI's own site, alongside a companion site, llmstxt.org. The problem it was built to solve: a language model's context window is too small to hold an entire website, and converting a page full of navigation, ads, and JavaScript into clean text for a model to read is slow and error-prone. The proposal specifies a /llms.txt file (a short markdown overview with links to further markdown documents) and an optional /llms-full.txt file (the entire documentation set concatenated into one file, for cases where a model needs the whole thing at once). It was explicitly framed around developer documentation and API references, not general content marketing.

llms.txt has never gone through a standards body. It's a community convention, tracked informally by llmstxt.org, not an IETF or W3C standard, and no search engine or AI lab has adopted it as a formal specification.

What a well-structured llms.txt file looks like

The original proposal, still hosted at llmstxt.org, specifies a fairly rigid shape, not a free-form text blob. The file opens with a single H1 heading naming the project or site, the only strictly required part of the whole spec. Directly under that comes a blockquote: a short summary, one or two sentences, containing whatever context a model needs to understand everything that follows. After the blockquote, the file can include any number of ordinary markdown sections (paragraphs, lists, anything except further headings) with more background. The bulk of the file then lives under H2-level headings, each one holding a plain markdown list of links in the form [Link title](https://link-url): optional description, grouped under whatever categories fit the site, documentation, guides, API reference, changelog, or similar.

One convention to know: the spec calls out an "Optional" H2 section specifically for secondary links, material a model can skip entirely when it's working with a smaller context window and doesn't need the full picture. A minimal, spec-compliant file looks roughly like this:

# Acme Docs

> Acme is a tool for X. This file indexes the documentation for AI systems that need a token-efficient overview.

## Docs

- [Getting Started](https://docs.acme.com/start): setup and first steps
- [API Reference](https://docs.acme.com/api): full endpoint list

## Optional

- [Changelog](https://docs.acme.com/changelog)

Nothing about this format requires special tooling to produce. It's plain markdown, which is the whole point: a model can parse it with the same reasoning it already applies to any other markdown document, without needing a dedicated parser built just for this file.

What does Google's own guidance say?

Google published an official guide, Optimizing your website for generative AI features on Google Search, in May 2026, and added a note on llms.txt in June 2026, addressing this question directly for its own AI Overviews and AI Mode. Its stated position is blunt, "Google Search ignores them," and in more detail: Google's crawlers may discover an llms.txt file if one exists, but it's treated like any other text file on the site, with no special indexing pathway and no effect on eligibility, prominence, or ranking inside AI Overviews or AI Mode. The same guide also states that manually chunking content into smaller pieces for AI systems isn't necessary, since Google's systems can identify the relevant part of a longer page without that extra work.

That guidance is specific to Google Search, including its generative AI features. It isn't a claim about every AI product. ChatGPT, Claude, Perplexity, and Gemini don't necessarily share Google's exact retrieval mechanics, and Google's own guide says as much: the statement applies to Google Search, not to browser agents, standalone AI assistants, or other operators' answer engines.

Lines of HTML code on a screen, representing the developer documentation llms.txt was originally designed to summarise for a model

Do other AI crawlers read it?

This is the part where the evidence runs out. As of 2026, no major AI operator (OpenAI, Anthropic, Google, Meta) has publicly stated that GPTBot, ClaudeBot, Google-Extended, or any other named crawler reads llms.txt and uses it to decide what to retrieve or cite. Absence of a public statement isn't proof a crawler ignores the file entirely, but it does mean the confident claims circulating in SEO content that llms.txt "improves AI citations" are not backed by anything an operator has confirmed. Treat any such claim, including specific percentage-lift numbers attached to it, as unverified marketing until an operator says otherwise.

What is independently verifiable: some AI companies publish llms.txt files for their own developer documentation. Anthropic's own developer docs, for example, expose both an llms.txt and a full llms-full.txt at their documentation domain. That confirms the format has real adoption among API and developer-tool companies publishing their own docs. It does not confirm that any crawler treats the file as a retrieval or ranking signal when reading someone else's site. Publishing an llms.txt and a crawler consuming it as a meaningful signal are two separate claims, and only the first one has solid evidence behind it.

How AI coding assistants use it: Cursor, Windsurf and Claude Code

The clearest, most verifiable adoption of llms.txt isn't on the search side at all. It's inside code editors, and even there the pattern is narrower than the marketing suggests. Cursor lets a user manually add third-party documentation through its Docs feature (@Docs in chat), and the modal for adding a new doc explicitly accepts a link to an /llms-full.txt file as one input option. That's a real, verifiable feature. What it isn't, on the available evidence, is automatic: the documented flow has a person point the editor at the file directly, and nothing suggests it crawls the web looking for llms.txt the way Googlebot crawls for robots.txt.

Windsurf, the Codeium-built editor, takes a more self-referential approach: its own documentation site publishes both an llms.txt and an llms-full.txt at docs.windsurf.com, the same pattern Anthropic and Cursor follow for their own docs. That confirms the format is useful for a documentation team publishing its own reference material in an AI-consumable shape. It says nothing about whether Windsurf's coding agent, Cascade, autonomously fetches other sites' llms.txt files in the background while it works, which isn't publicly documented behaviour anywhere.

Claude Code follows a related but distinct pattern: it fetches a specific, known documentation URL only when a task calls for it, through an explicit web-fetch tool, rather than passively ingesting a background llms.txt index on every run. Across all three tools, the common thread is the same: llms.txt earns its keep as something a person or an agent deliberately points at for a specific documentation task, not something ambient that boosts a site's general visibility just by existing.

Where llms.txt helps right now

Use caseDoes llms.txt help?
Improving AI Overview or AI Mode visibility on GoogleNo, per Google's own guidance (May 2026, clarified June 2026)
Getting cited more often by ChatGPT, Claude, or Perplexity's answer featuresUnconfirmed, no operator has stated this
An AI coding assistant reading your API docs directly (for example, inside an integrated development environment (IDE) or agent workflow)Plausibly yes, this is the original use case the format was designed for
Giving a model a fast, token-efficient index instead of parsing full HTML navigation and adsPlausibly yes, for documentation-heavy sites specifically
Replacing solid crawl access, clean HTML, and real E-E-A-T (experience, expertise, authoritativeness and trust) signalsNo, and nothing suggests it's a substitute; see do AI crawlers read your site for what gates access

The pattern across every credible source on this: llms.txt was built for, and still mainly helps, developer-documentation and API-reference content read by coding assistants and agent tooling. It was never built as, and hasn't been shown to function as, a general AI-search-visibility lever for ordinary content pages.

The llms-full.txt variant: when is it worth building?

The original proposal also defines a second, optional file: llms-full.txt, the entire documentation set concatenated into one markdown file instead of an index of links. Anthropic and Windsurf both publish one alongside their standard llms.txt, and it's the file Cursor's Docs modal accepts directly, which is the strongest evidence that this specific variant gets consumed by real tooling rather than sitting unused.

The tradeoff is size. A small site's llms-full.txt might run a few hundred lines and cost a model almost nothing to read. A large API reference or a documentation set with hundreds of pages can produce a file running into the tens of thousands of lines, which starts to work against the exact problem llms.txt was built to solve: giving a model an efficient way into the content instead of an expensive one. There's no fixed threshold at which it stops being worth it, but the practical test is simple: if a human wouldn't want to open the file and scroll through it, a model working inside a limited context window probably doesn't benefit from having the whole thing loaded either. For a documentation-heavy site, building the standard llms.txt index first and adding llms-full.txt only if it stays a reasonable size is the safer order to do the work in.

Silhouette of a road signpost at golden hour, representing the decision of whether llms.txt is worth adding to a given site

Should you add one to your site?

There's effectively no downside to serving an llms.txt file. It costs nothing to host and can't hurt a normal page's crawlability, since it lives at its own URL and doesn't touch anything else. The real question is whether it's worth the time to build and maintain one.

For a documentation or API-reference site, the case is reasonable: a curated, link-rich markdown summary reduces the work a coding-assistant model has to do, and it fits directly inside the original purpose Howard designed it for. For everything else, whether a useful piece of content becomes visible to AI systems depends far more on the fundamentals: whether AI crawlers can reach and render the page at all (see robots.txt and AI crawlers), and whether the content itself states a clear, citable answer (see how to show up in AI search results). An llms.txt file sitting on top of a site that AI crawlers can't access, or content that never states a direct answer, changes nothing. Access and clarity come first; llms.txt, if it belongs on a site at all, is a minor addition after those are already solid.

Related: AEO, GEO and SEO: what each term means, do AI crawlers read your site, robots.txt and AI crawlers.

Frequently asked questions

Does adding an llms.txt file hurt anything?

No. It's a plain text file that sits at the site root and costs nothing to serve. The risk isn't harm, it's opportunity cost: time spent maintaining a curated llms.txt instead of fixing crawl access or content quality, which is what gates AI-search visibility.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers which URLs they're allowed to fetch. llms.txt is a proposed curated index (a markdown summary with links) meant to help a language model orient itself in a site's content. One is a permissions file; the other is a navigation aid, and neither is required by any major search or AI operator.

Should I still add llms.txt to my site?

There's no evidence it moves AI Overviews, AI Mode, or general AI-citation visibility, per Google's own guidance (published May 2026, with an llms.txt note added in June 2026) and the lack of any public confirmation from OpenAI, Anthropic, or Meta that their crawlers use it as a retrieval signal. If your site serves developer documentation that AI coding assistants read directly, it's worth adding as a low-cost convenience. For a general content site, fixing crawl access and content quality (see robots.txt and AI crawlers) matters far more.

Do Cursor, Windsurf, or Claude Code automatically crawl a site's llms.txt file?

No, not automatically. Cursor's Docs feature accepts an llms.txt or llms-full.txt URL only when a person manually adds it. Windsurf and Claude Code follow the same manual, on-demand pattern rather than passively crawling llms.txt files in the background.

When is it worth building an llms-full.txt instead of just llms.txt?

When the site is documentation-heavy enough that a single concatenated file stays a reasonable size, roughly the point where a human could still skim it. For a small site it adds little over the standard index; for a large API reference, it gives a coding assistant one high-signal file instead of dozens of separate fetches.