llms.txt Explained: What It Does, Who Uses It, and What It Does Not Do

llms.txt is a proposed Markdown discovery file for publishing a curated list of important pages. Support remains incomplete, Google does not use it, and the file does not guarantee citations. It is optional, cheap to publish, and best treated as a small discovery aid after crawlability, retrieval access, schema, authorship, and substantive content are in place.

This guide is the technical walkthrough we wish existed when we started building llms.txt files for client sites. What it is, where it came from, what to put in it, how it differs from the files you already have, and how to validate it. If you want the broader strategic context first, read our pillar guide on what AEO actually is. Otherwise, let's get into the file itself.

What llms.txt Is, Exactly

llms.txt is a Markdown file published at yoursite.com/llms.txt. It offers compatible tools a hand-curated summary of important pages. It does not replace crawling, control access, or tell an engine what to cite.

The format is opinionated. An H1 with the site or product name. A short blockquote summarizing what the site is. Then a series of H2 sections (Docs, Pricing, Case studies, About, FAQ) that group your most important URLs, each with a one-line description. The whole file is usually 50 to 200 lines.

The point is curation, not coverage. sitemap.xml lists discoverable URLs. llms.txt can summarize a smaller set of important pages for tools that choose to read it. Support is incomplete, so the file should not be treated as a citation directive.

Where llms.txt Came From

The proposal came from Jeremy Howard (co-founder of fast.ai and Answer.AI) in September 2024. The motivation was simple. Reasoning models were burning tokens trying to scrape and parse arbitrary websites, and the most useful pages on most sites were buried in nav menus, JavaScript, and analytics tags. A simple, predictable file at a known location would let models skip the noise.

llms.txt remains a proposed convention, not a ratified web standard. Some tools and documentation systems use it, but support is incomplete. Google has stated that it does not use llms.txt. Publish the file only as an optional discovery aid, not as required infrastructure.

The File Format, Step by Step

llms.txt is intentionally simple Markdown. Here is a minimal example for a SaaS product:

That is the whole format. An H1, a blockquote, H2 sections, bulleted links with one-line descriptions. The proposal reserves an Optional H2 for secondary links. A compatible tool may ignore that section or the entire file.

How llms.txt Differs From robots.txt and sitemap.xml

This is where most teams get confused. The three files do different jobs. robots.txt controls access, sitemap.xml supports URL discovery, and llms.txt is an optional summary for compatible tools.

robots.txt: The Permissions File

robots.txt tells crawlers what they can and cannot access. It does not describe content or rank pages. For retrieval, check OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, and PerplexityBot. GPTBot and Google-Extended are training controls, so their policy is a separate decision.

sitemap.xml: The Discovery File

sitemap.xml is the comprehensive list of every URL on your site. It exists so crawlers (search and AI alike) can find every page without relying on internal links. It tells crawlers what exists. It does not tell them what matters.

llms.txt: The Curation File

llms.txt is an optional editorial summary. It can state what a site is and link to selected pages with short descriptions. Compatible tools may use that context, but the file does not set page priority or determine citations.

If robots.txt is the door and sitemap.xml is the table of contents, llms.txt is the executive summary.

What to Put in llms.txt for Different Site Types

SaaS or Product Sites

Documentation is often the most useful section because it answers product questions directly. If you include docs, link only to stable, public pages and describe each one accurately.

Editorial or Publisher Sites

Lead with what the publication covers, then sections for major topic areas, author pages, and About. Author pages provide readers and retrieval systems with accountable authorship. They do not guarantee that an article will be cited.

Agency or Service Business Sites

Case studies can provide concrete, verifiable proof of work. Include the strongest public examples and describe what each page documents.

Ecommerce Sites

Lead with what the store sells and the brand promise, then sections for Top Categories, Bestsellers, Brand Story, and Customer Support. Skip the long tail of product URLs. The point is to give compatible tools concise brand context, not a copy of the catalog.

The Common Mistakes

Treating It Like sitemap.xml

The biggest mistake is dumping every URL on the site into llms.txt. The whole point is curation. If you have 4,000 pages, llms.txt should reference maybe 30 of them, organized into clear sections. The rest belong in sitemap.xml where they always belonged.

Skipping the Descriptions

One-line descriptions make the file clearer. "Pricing" says little. "Pricing: tiered SaaS pricing with annual and monthly options, free trial available" tells a compatible tool what the linked page contains. It does not determine whether the page gets crawled or cited.

Stale Links

If you publish llms.txt, keep its links current. Remove or update retired URLs as part of the same maintenance used for navigation and sitemaps.

Wrong Content Type

The file must be served as text/plain or text/markdown. Some hosting setups serve unknown files as application/octet-stream. Check the response headers and confirm the file is readable as plain text or Markdown.

Forgetting the Optional Section

The proposal uses the Optional H2 for secondary links. Put lower-priority resources there only if that structure makes the file clearer. Tool support and interpretation vary.

How to Validate Your llms.txt

There is no formal validator yet but the practical checklist is short:

• The file is reachable at exactly /llms.txt at the site root, not in a subfolder.
• It is served with content-type text/plain or text/markdown.
• It is valid Markdown (the Markdown parser of your choice doesn't choke).
• It starts with an H1 followed by a blockquote.
• Every linked URL returns a 200 status.
• The file is under 200 lines (longer is allowed but signals you are dumping rather than curating).

The Meridian15 AEO audit treats llms.txt as one optional discovery check worth 2 points out of 66. The check confirms that the root file is reachable. The remaining checks cover retrieval access, schema, authorship, headings, metadata, sitemap quality, crawlable content, and breadcrumbs. We explain the full scoring contract in this companion post.

How to Ship It

Three steps. First, draft the file by hand. Do not generate it from a sitemap dump, because the value is in the curation. Second, drop the file at the root of your domain (next to robots.txt and sitemap.xml). Most static hosts (Netlify, Vercel, Cloudflare Pages) just need it in your public folder. Third, validate it with our free AEO readiness check to confirm it is reachable, well-formed, and serving the right content type.

Treat the file as maintained content. Update it when a linked page moves or when a more useful public resource replaces an older one.

Do not attribute citation changes to llms.txt alone. Track a stable set of branded and category queries across the engines you care about, then record source changes alongside broader site and authority work. We compared Perplexity tracking options here.

When llms.txt Is Worth Publishing

The case for llms.txt is modest: it is cheap to publish and may help compatible tools discover selected pages. There is no reliable adoption benchmark or evidence that early publication creates a citation advantage.

llms.txt is not comparable to HTTPS or mobile responsiveness. Those are established requirements. llms.txt remains optional, with incomplete support and no proven citation advantage.

If you want the broader playbook (structured data, FAQ schema, named authors, citation-worthy content patterns), read our pillar guide on AEO. If you just want to know whether your site is set up correctly, run it through our AEO audit tool. Both are free.

Frequently Asked Questions

What is llms.txt and where does it live?

llms.txt is a proposed Markdown discovery file published at yoursite.com/llms.txt. It can summarize important public pages for tools that choose to read it. Support is incomplete, and the file does not control crawling or guarantee citations.

Is llms.txt an official standard?

No. llms.txt is a proposed convention, not a ratified web standard. Support is incomplete, and Google does not use it.

How is llms.txt different from robots.txt and sitemap.xml?

robots.txt controls crawler access. sitemap.xml lists URLs for discovery. llms.txt is an optional curated summary for compatible tools. It does not replace either file.

What should I put in my llms.txt file?

Start with an H1 naming the site, a short blockquote that describes it, and H2 sections linking to a small set of important public pages. Give each link a factual one-line description. Keep the file curated and current.

Will llms.txt actually get me cited in AI answers?

No. Google does not use llms.txt, support elsewhere is incomplete, and the file does not guarantee citations. Treat it as optional discovery support after crawlable content, retrieval access, schema, and authorship are in place.

How do I validate my llms.txt file?

Check that the root file is reachable, valid Markdown, and links to live public pages. The Meridian15 audit treats llms.txt as an optional 2-point check out of 66.