Live on Product Hunt Intraface launched today. Less steering, more vacations. See the launch
Start for free

What is llms.txt?

llms.txt is a proposed plain-text file, served at /llms.txt, that gives AI agents a short Markdown guide to a website: a title, a one-paragraph summary, and lists of links with notes. Jeremy Howard proposed it in September 2024. Thousands of sites publish one, but no major search or answer engine has documented reading the file.

Part of Glossary: the terms behind MCP, coding agents and shared pages, defined plainly.

How it works

An agent fetches the small llms.txt file first, then follows only the links it needs instead of crawling the whole site.at the site rootideally as MarkdownAI agenta small context window/llms.txtthe pages it links1. fetches one file2. summary and links3. reads what it needs
  1. The site publishes a Markdown file at /llms.txt: an H1 with the site's name, a blockquote summary, then H2 sections of links, each with a short note.

  2. An agent that needs to know about the site fetches that one file first. It is small, so it fits in a context window where the whole site would not.

  3. Each link points at a page, ideally a Markdown copy of it (the same URL with .md appended), so the agent reads clean text instead of navigation and scripts.

  4. An H2 called Optional, at the end, holds secondary links an agent can skip when it is short on context.

  5. Nothing obliges an engine to read the file. Whether ChatGPT, Claude, Perplexity or Google fetch it is not documented by any of them.

Source: The /llms.txt file proposal (llmstxt.org)

llms.txt versus robots.txt and sitemap.xml

The three files sit at the site root and get confused because of it. robots.txt is about permission: which user agents may fetch which paths. sitemap.xml is about completeness: every URL you want indexed, with a last-modified date, and nothing about what any URL means. llms.txt is about meaning. It is written for a reader with a limited context window, so it is curated rather than complete, and every link carries a note saying what the page is for.

The format is deliberately small. The proposal at llmstxt.org requires only an H1 with the site's name. After that come an optional blockquote summary, any paragraphs you need, H2 sections whose list items are [name](url): note, and an H2 called Optional for links an agent can drop when it is short on space. The proposal also asks sites to serve a Markdown version of each linked page at the same URL with .md appended, or with the extension swapped, and to advertise it with a rel="alternate" type="text/markdown" link.

Example: intraface.cc/llms.txt

Intraface publishes one at https://intraface.cc/llms.txt. It opens with # Intraface, then a blockquote that says what the product is in one paragraph. The H2 sections follow the site's shape: how it works, the concepts, a section for agents with the MCP address and CLI commands, then one list per page type (guides, integrations, glossary, comparisons, alternatives) where each line is the page's title, its link and a one-sentence summary. The glossary section alone tells an agent what every term page defines, which is more than the sitemap at https://intraface.cc/sitemap.xml can say about the same URLs.

The file is generated from the same content files as the pages, so a new page appears in both places at once and the summaries never drift from the descriptions on the pages themselves. That is the detail most hand-written files get wrong: they are accurate on the day they are written and stale a month later.

What is known about who reads it

Be careful with the adoption numbers. The proposal's own page says thousands of sites publish an llms.txt, that documentation platforms generate one automatically, that Chrome's Lighthouse audits for it, and that OpenAI, Anthropic and Gemini publish llms.txt files for their developer docs. All of that is about publishing. None of it says a crawler or an assistant consumes the file, and none of the engines' own crawler documentation mentions it. Google's guidance goes further and says you don't need any new machine readable files to appear in its AI features.

So the honest position today: an agent that is pointed at your site, or that follows the convention on its own, gets a better first read. A search engine deciding whether to cite you is not known to look.

Common mistakes

  • Treating it as a ranking lever. It is a courtesy to agents, not a signal. Spend the effort on the pages it links first.
  • Linking to HTML with no Markdown twin. The agent then has to parse your navigation, cookie banner and scripts to find the text. The proposal's .md convention exists to avoid that.
  • Writing a brochure. The blockquote and notes are for a machine deciding what to open next. Say what each page contains, not why your product is great.
  • Letting it go stale. Generate it from your content, or add it to the same checklist as the sitemap.

FAQ

Does Google read llms.txt?
Google has not said so. Its guidance on AI features says there are no extra requirements to appear in AI Overviews or AI Mode and that you don't need to create new machine readable files or AI text files. It does not mention llms.txt at all. Treat the file as unread by Google unless Google says otherwise.
What is the difference between llms.txt and robots.txt?
robots.txt tells crawlers which paths they may fetch. llms.txt tells an agent what the site is and which pages matter. One is a permission file, the other a reading list. The proposal also says sitemap.xml is not a substitute, because a sitemap lists every URL without saying what any of them contains.
Does an llms.txt file help SEO?
There is no evidence that it does. No search engine lists it as a signal, and the pages it links still have to be crawlable and indexed on their own. Publish it because it is cheap and useful to agents that do fetch it, not because it moves rankings.

Stop sending HTMLover Slack.

Create an account in a minute, connect your coding agent, and make your first board.