llms.txt

The convention was proposed in September 2024 and is documented at llmstxt.org. The file is markdown rather than XML: an H1 with the site name, a blockquote summary, then sections of links with one line of context each. A companion file, llms-full.txt, holds the full text of those pages in one fetch so an agent does not have to crawl and strip HTML to read the documentation.

It is a convention, not a ratified standard, and no major AI provider documents reading it. That is the honest state of the evidence as of July 2026. Publishing one is still cheap, and the exercise of writing it forces a site to state plainly what it is and which pages matter, which is the part that helps whether or not a crawler ever fetches the file.

For AI readiness, the file existing is only half the check. A file that returns 200 but lists URLs that 404 is worse than no file at all, because it sends an agent that did the right thing to a dead end. Checked 2026-07-30 across five AI-visibility vendors: one served an llms.txt in which the blog links returned 404 because they omitted a path segment, and one advertised the file in both robots.txt and its sitemap while the URL itself returned a 404 page. That is why our readiness audit resolves every link in the file rather than checking that the file exists.

Frequently asked questions

Is llms.txt the same as robots.txt?

No. robots.txt states which crawlers may fetch what, and every major crawler reads it. llms.txt suggests what a language model should read first, and no major provider commits to reading it.

Do AI engines actually use llms.txt?

No major provider documents using it as of July 2026. Treat it as a low-cost signal and a useful writing exercise, not as a route into AI answers.

How do I check my llms.txt is valid?

Fetch it, confirm it returns 200 with a text content type, then request every URL it lists. Dead links inside the file are the most common defect we see, ahead of formatting mistakes.