llms.txt explained: what it does, who uses it, and what it does not do
llms.txt is a proposed Markdown file at yoursite.com/llms.txt that lists a curated set of your most important pages, with a title, a short summary, sections and one-line link descriptions. It is not a ratified standard, support is incomplete, Google Search ignores it, and it does not guarantee citations. robots.txt controls access and sitemap.xml lists URLs; llms.txt is an optional summary, worth publishing only after crawler access, schema and content are in place.
llms.txt is a proposed Markdown discovery file for publishing a curated list of important pages. Support remains incomplete, Google does not use it, and the file does not guarantee citations. It is optional, cheap to publish, and best treated as a small discovery aid after crawlability, retrieval access, schema, authorship, and substantive content are in place.
This guide is the technical walkthrough we wish existed when we started building llms.txt files for client sites. What it is, where it came from, what to put in it, how it differs from the files you already have, and how to validate it. If you want the broader strategic context first, read our pillar guide on what AEO actually is. Otherwise, let’s get into the file itself.
What llms.txt Is, Exactly
llms.txt is a Markdown file published at yoursite.com/llms.txt. It offers compatible tools a hand-curated summary of important pages. It does not replace crawling, control access, or tell an engine what to cite.
The format is opinionated. An H1 with the site or product name. A short blockquote summarizing what the site is. Then a series of H2 sections (Docs, Pricing, Case studies, About, FAQ) that group your most important URLs, each with a one-line description. Only the H1 is required. The proposal sets no length limit; it asks that the file stay small enough to fit in context, with the detail behind the links.
The point is curation, not coverage. sitemap.xml lists discoverable URLs. llms.txt can summarize a smaller set of important pages for tools that choose to read it. Support is incomplete, so the file should not be treated as a citation directive.
Where llms.txt came from
Jeremy Howard (co-founder of fast.ai and Answer.AI) published the llms.txt proposal on September 3, 2024. It gives two reasons: context windows are still too small to hold most websites, and turning HTML pages full of navigation, ads, and JavaScript back into clean text is difficult and imprecise. A simple, predictable file at a known location lets a tool skip that noise.
llms.txt remains a proposed convention, not a ratified web standard. Some tools and documentation systems use it, but support is incomplete. Google says Google Search ignores llms.txt files, and that publishing one will neither help nor harm a site in Google Search. Publish the file only as an optional discovery aid, not as required infrastructure.
The file format, step by step
llms.txt is intentionally simple Markdown. Here is a minimal example for an agency site like ours:
That is the whole format. An H1, a blockquote, H2 sections, bulleted links with one-line descriptions. The proposal reserves an Optional H2 for secondary links. A compatible tool may ignore that section or the entire file.
How llms.txt Differs From robots.txt and sitemap.xml
This is where most teams get confused. The three files do different jobs. robots.txt controls access, sitemap.xml supports URL discovery, and llms.txt is an optional summary for compatible tools.
robots.txt: The Permissions File
robots.txt tells crawlers what they can and cannot access. It does not describe content or rank pages. For retrieval, check OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, and PerplexityBot. GPTBot and Google-Extended are training controls, so their policy is a separate decision.
sitemap.xml: The Discovery File
sitemap.xml is the comprehensive list of every URL on your site. It exists so crawlers (search and AI alike) can find every page without relying on internal links. It tells crawlers what exists. It does not tell them what matters.
llms.txt: The Curation File
llms.txt is an optional editorial summary. It can state what a site is and link to selected pages with short descriptions. Compatible tools may use that context, but the file does not set page priority or determine citations.
If robots.txt is the door and sitemap.xml is the table of contents, llms.txt is the executive summary.
What to put in llms.txt for different site types
SaaS or Product Sites
Documentation is often the most useful section because it answers product questions directly. If you include docs, link only to stable, public pages and describe each one accurately.
Editorial or Publisher Sites
Lead with what the publication covers, then sections for major topic areas, author pages, and About. Author pages provide readers and retrieval systems with accountable authorship. They do not guarantee that an article will be cited.
Agency or Service Business Sites
Case studies can provide concrete, verifiable proof of work. Include the strongest public examples and describe what each page documents.
Ecommerce Sites
Lead with what the store sells and the brand promise, then sections for Top Categories, Bestsellers, Brand Story, and Customer Support. Skip the long tail of product URLs. The point is to give compatible tools concise brand context, not a copy of the catalog.
The common mistakes
Treating It Like sitemap.xml
Choose pages that help a compatible tool understand the site. Keep the descriptions useful and the links current. There is no universal URL count to target; sitemap.xml remains the place for the broader list of discoverable pages.
Skipping the Descriptions
One-line descriptions make the file clearer. “Pricing” says little. “Pricing: tiered SaaS pricing with annual and monthly options, free trial available” tells a compatible tool what the linked page contains. It does not determine whether the page gets crawled or cited.
Stale Links
If you publish llms.txt, keep its links current. Remove or update retired URLs as part of the same maintenance used for navigation and sitemaps.
Wrong Content Type
The proposal does not specify a content type. We serve it as text/plain or text/markdown, because some hosting setups serve unknown files as application/octet-stream. Check the response headers and confirm the file is readable as plain text or Markdown.
Forgetting the Optional Section
The proposal uses the Optional H2 for secondary links. Put lower-priority resources there only if that structure makes the file clearer. Tool support and interpretation vary.
How to validate your llms.txt
There is no formal validator yet but the practical checklist is short:
- The file is reachable at exactly /llms.txt at the site root, not in a subfolder.
- It is served with content-type text/plain or text/markdown (our recommendation, not a rule in the proposal).
- It is valid Markdown (the Markdown parser of your choice doesn’t choke).
- It starts with an H1 followed by a blockquote.
- Every linked URL returns a 200 status.
- The file is easy to scan, with useful descriptions and no duplicate or retired links.
The Meridian15 AEO audit treats llms.txt as one optional discovery check worth 2 points out of 66. The check looks for a reachable root file, an H1, and at least two Markdown links. The remaining checks cover retrieval access, schema, authorship, headings, metadata, sitemap quality, crawlable content, and breadcrumbs. We explain the full scoring contract in this companion post.
How to ship it
Three steps. First, draft the file by hand. Do not generate it from a sitemap dump, because the value is in the curation. Second, drop the file at the root of your domain (next to robots.txt and sitemap.xml). Most static hosts (Netlify, Vercel, Cloudflare Pages) just need it in your public folder. Third, use our free AEO readiness check to check reachability and basic heading and link structure. Check that its linked pages are live and its descriptions are accurate.
Treat the file as maintained content. Update it when a linked page moves or when a more useful public resource replaces an older one.
Do not attribute citation changes to llms.txt alone. Track a stable set of branded and category queries across the engines you care about, then record source changes alongside broader site and authority work. We compared Perplexity tracking options here.
When llms.txt is worth publishing
The case for llms.txt is modest: it is cheap to publish and may help compatible tools discover selected pages. There is no reliable adoption benchmark or evidence that early publication creates a citation advantage.
llms.txt is not comparable to HTTPS or mobile responsiveness. Those are established requirements. llms.txt remains optional, with incomplete support and no proven citation advantage.
If you want the broader playbook (structured data, FAQ schema, named authors, citation-worthy content patterns), read our pillar guide on AEO. If you just want to know whether your site is set up correctly, run it through our AEO audit tool. Both are free.
Questions, answered.
What is llms.txt and where does it live?
llms.txt is a proposed Markdown discovery file published at yoursite.com/llms.txt. It can summarize important public pages for tools that choose to read it. Support is incomplete, and the file does not control crawling or guarantee citations.
Is llms.txt an official standard?
No. llms.txt is a proposed convention, not a ratified web standard. Support is incomplete, and Google Search ignores it.
How is llms.txt different from robots.txt and sitemap.xml?
robots.txt controls crawler access. sitemap.xml lists URLs for discovery. llms.txt is an optional curated summary for compatible tools. It does not replace either file.
What should I put in my llms.txt file?
Start with an H1 naming the site, a short blockquote that describes it, and H2 sections linking to a small set of important public pages. Give each link a factual one-line description. Keep the file curated and current.
Will llms.txt actually get me cited in AI answers?
No. Google Search ignores llms.txt, support elsewhere is incomplete, and the file does not guarantee citations. Treat it as optional discovery support after crawlable content, retrieval access, schema, and authorship are in place.
How do I validate my llms.txt file?
Check that the root file is reachable, valid Markdown, and links to live public pages. The Meridian15 audit treats llms.txt as an optional 2-point check out of 66.
Audit your site’s technical AEO foundation.
Run any URL through the Meridian15 AEO audit. We score 13 technical checks including optional llms.txt discovery, schema, retrieval access, and crawlable content. See the full SEO and AEO service.
