Does llms.txt Actually Work for AI Search?

llms.txt is a markdown file, served at a site's root URL, that proposes a curated summary of a site's content plus links to markdown versions of key pages, meant to help a language model orient itself without parsing full HTML. As of Google's official May 2026 guidance, it isn't required or specially treated for AI Overviews or AI Mode, and no major AI operator (OpenAI, Anthropic, Google, Meta) has publicly confirmed using it as a ranking or retrieval signal in production. It's a genuinely useful convenience in a narrower case: AI coding assistants reading developer documentation directly.

Where Did llms.txt Come From?

Jeremy Howard, co-founder of Answer.AI and fast.ai, published the original proposal on September 3, 2024, on Answer.AI's own site, alongside a companion site, llmstxt.org, that catalogs adopters. The problem it was built to solve: a language model's context window is too small to hold an entire website, and converting a page full of navigation, ads, and JavaScript into clean text for a model to read is slow and error-prone. The proposal specifies a /llms.txt file (a short markdown overview with links to further markdown documents) and an optional /llms-full.txt file (the entire documentation set concatenated into one file, for cases where a model needs the whole thing at once). It was explicitly framed around developer documentation and API references, not general content marketing.

llms.txt has never gone through a standards body. It's a community convention, tracked informally by llmstxt.org, not an IETF or W3C standard, and no search engine or AI lab has adopted it as a formal specification.

What a Well-Structured llms.txt File Actually Looks Like

The original proposal, still hosted at llmstxt.org, specifies a fairly rigid shape, not a free-form text blob. The file opens with a single H1 heading naming the project or site, the only strictly required part of the whole spec. Directly under that comes a blockquote: a short summary, one or two sentences, containing whatever context a model needs to understand everything that follows. After the blockquote, the file can include any number of ordinary markdown sections (paragraphs, lists, anything except further headings) with more background. The bulk of the file then lives under H2-level headings, each one holding a plain markdown list of links in the form [Link title](https://link-url): optional description, grouped under whatever categories fit the site, documentation, guides, API reference, changelog, or similar.

One convention worth knowing: the spec calls out an "Optional" H2 section specifically for secondary links, material a model can skip entirely when it's working with a smaller context window and doesn't need the full picture. A minimal, spec-compliant file looks roughly like this:

# Acme Docs

> Acme is a tool for X. This file indexes the documentation for AI systems that need a token-efficient overview.

## Docs

- [Getting Started](https://docs.acme.com/start): setup and first steps
- [API Reference](https://docs.acme.com/api): full endpoint list

## Optional

- [Changelog](https://docs.acme.com/changelog)

Nothing about this format requires special tooling to produce. It's plain markdown, which is the whole point: a model can parse it with the same reasoning it already applies to any other markdown document, without needing a dedicated parser built just for this file.

What Does Google's Own Guidance Say?

Google published an official guide, Optimizing your website for generative AI features on Google Search, in May 2026, addressing this question directly for its own AI Overviews and AI Mode. Its stated position: Google's crawlers may discover an llms.txt file if one exists, but it's treated like any other text file on the site, with no special indexing pathway and no effect on eligibility, prominence, or ranking inside AI Overviews or AI Mode. The same guide also states that manually chunking content into smaller pieces for AI systems isn't necessary, since Google's systems can identify the relevant part of a longer page without that extra work.

That guidance is specific to Google Search, including its generative AI features. It isn't a claim about every AI product. ChatGPT, Claude, Perplexity, and Gemini don't necessarily share Google's exact retrieval mechanics, and Google's own guide says as much: the statement applies to Google Search, not to browser agents, standalone AI assistants, or other operators' answer engines.

Do Other AI Crawlers Actually Read It?

This is the part where the evidence runs out. As of 2026, no major AI operator (OpenAI, Anthropic, Google, Meta) has publicly stated that GPTBot, ClaudeBot, Google-Extended, or any other named crawler reads llms.txt and uses it to decide what to retrieve or cite. Absence of a public statement isn't proof a crawler ignores the file entirely, but it does mean the confident claims circulating in SEO content that llms.txt "improves AI citations" are not backed by anything an operator has actually confirmed. Treat any such claim, including specific percentage-lift numbers attached to it, as unverified marketing until an operator says otherwise.

What is independently verifiable: some AI companies publish llms.txt files for their own developer documentation. Anthropic's own developer docs, for example, expose both an llms.txt and a full llms-full.txt at their documentation domain. That confirms the format has real adoption among API and developer-tool companies publishing their own docs. It does not confirm that any crawler treats the file as a retrieval or ranking signal when reading someone else's site. Publishing an llms.txt and a crawler consuming it as a meaningful signal are two separate claims, and only the first one has solid evidence behind it.

How AI Coding Assistants Actually Use It: Cursor, Windsurf, and Claude Code

The clearest, most verifiable adoption of llms.txt isn't happening on the search side at all, it's happening inside code editors, and even there the pattern is narrower than the marketing around it suggests. Cursor lets a user manually add third-party documentation through its Docs feature (@Docs in chat), and the modal for adding a new doc explicitly accepts a link to an /llms-full.txt file as one input option. That's a real, verifiable feature. What it isn't is automatic: threads on Cursor's own community forum describe attempts to have Cursor auto-discover and index a site's llms.txt file failing, with indexing only working once the URL is added by hand. The file has to be pointed at directly by a person; nothing crawls the web looking for it the way Googlebot crawls for robots.txt.

Windsurf, the Codeium-built editor, takes a more self-referential approach: its own documentation site publishes both an llms.txt and an llms-full.txt at docs.windsurf.com, the same pattern Anthropic and Cursor follow for their own docs. That confirms the format is genuinely useful for a documentation team publishing its own reference material in an AI-consumable shape. It says nothing about whether Windsurf's coding agent, Cascade, autonomously fetches other sites' llms.txt files in the background while it works, which isn't publicly documented behavior anywhere.

Claude Code follows a related but distinct pattern: it fetches a specific, known documentation URL only when a task calls for it, through an explicit web-fetch tool, rather than passively ingesting a background llms.txt index on every run. Anthropic's own Claude Code documentation ships a similar docs-map file for the same reason, a curated index of its own reference pages. Across all three tools, the common thread is the same: llms.txt earns its keep as something a person or an agent deliberately points at for a specific documentation task, not something ambient that boosts a site's general visibility just by existing.

Where llms.txt Actually Helps Right Now

The pattern across every credible source on this: llms.txt was built for, and still mainly helps, developer-documentation and API-reference content read by coding assistants and agent tooling. It was never built as, and hasn't been shown to function as, a general AI-search-visibility lever for ordinary content pages.

The llms-full.txt Variant: When Is It Worth Building

The original proposal also defines a second, optional file: llms-full.txt, the entire documentation set concatenated into one markdown file instead of an index of links. Anthropic and Windsurf both publish one alongside their standard llms.txt, and it's the file Cursor's Docs modal accepts directly, which is the strongest evidence that this specific variant gets consumed by real tooling rather than sitting unused.

The tradeoff is size. A small site's llms-full.txt might run a few hundred lines and cost a model almost nothing to read. A large API reference or a documentation set with hundreds of pages can produce a file running into the tens of thousands of lines, which starts to work against the exact problem llms.txt was built to solve: giving a model an efficient way into the content instead of an expensive one. There's no fixed threshold at which it stops being worth it, but the practical test is simple: if a human wouldn't want to open the file and scroll through it, a model working inside a limited context window probably doesn't benefit from having the whole thing loaded either. For a documentation-heavy site, building the standard llms.txt index first and adding llms-full.txt only if it stays a reasonable size is the safer order to do the work in.

Should You Add One to Your Site?

There's effectively no downside to serving an llms.txt file. It costs nothing to host and can't hurt a normal page's crawlability, since it lives at its own URL and doesn't touch anything else. The real question is whether it's worth the time to build and maintain one.

For a documentation or API-reference site, the case is reasonable: a curated, link-rich markdown summary genuinely reduces the work a coding-assistant model has to do, and it fits directly inside the original purpose Howard designed it for. For everything else, whether a genuinely useful piece of content becomes visible to AI systems depends far more on the fundamentals: whether AI crawlers can reach and render the page at all (see robots.txt and AI crawlers), and whether the content itself states a clear, citable answer (see how to show up in AI search results). An llms.txt file sitting on top of a site that AI crawlers can't actually access, or content that never states a direct answer, changes nothing. Access and clarity come first; llms.txt, if it belongs on a site at all, is a minor addition after those are already solid.

Related: AEO, GEO, and SEO: what each term actually means, do AI crawlers actually read your site, robots.txt and AI crawlers.