Guides
What is llms.txt, and do you need one?
Learn what llms.txt is, when it is worth publishing, and how to create a small maintained file that points agents to useful site content.
llms.txt is a proposed Markdown file that gives AI agents a concise overview of a site and links to useful content.
The idea is simple: instead of making an agent discover and interpret an entire website from scratch, you give it a small, structured starting point.
An llms.txt file can be useful when you have important pages worth highlighting and someone can keep the file current.
It is not a ranking signal or a guarantee that an AI system will use, cite, or prefer the pages you list.
What goes in an llms.txt file
The llms.txt proposal defines a simple Markdown format.
A file can include:
- an H1 with the site or project name
- a short summary
- optional context about the site
- sections containing links to useful resources
- short descriptions that explain what each resource contains
For example:
# Northline CRM
> CRM software for service teams.
## Product
- [Product overview](https://example.com/product): What Northline CRM does and who it is for.
- [Pricing](https://example.com/pricing): Current plans and pricing.
## Resources
- [Documentation](https://docs.example.com/): Setup guides and product reference.
- [Integrations](https://example.com/integrations): Supported integrations and setup information.
- [Support](https://example.com/support): Troubleshooting and support options.
Keep the file concise. It should help an agent understand where useful information lives, not reproduce the whole site.
Publish one when you can maintain it
An llms.txt file is most useful when someone owns the pages it references and updates the file when those pages change.
Before publishing one, make sure the team can:
- choose the pages that matter
- write useful descriptions for those pages
- remove or update stale links
- recheck the file after important product, pricing, or documentation changes
If nobody will maintain it, wait.
A short current file is more useful than a large index full of retired or misleading links.
Choose the pages that matter most
Do not copy your entire sitemap into llms.txt.
Start with the pages that explain the product or help users complete important tasks.
For a software company, that might include:
- product overview
- pricing
- documentation
- API reference
- integrations
- support
- important comparison or use-case pages
For technical documentation, the file might instead highlight a quickstart, API reference, authentication guide, and a few important tutorials.
The right list depends on what you want an agent to understand about the site.
Publish it at the right path
Use /llms.txt when the file describes the whole site.
You can also publish a file under a subpath, such as:
/docs/llms.txt
That file can describe content under /docs/.
This is useful when different parts of a site have different owners or purposes.
After publishing the file:
- fetch the deployed
llms.txt - open the links it contains
- check that the descriptions still match the pages
- remove stale or redirected URLs
- repeat the check after meaningful site changes
The deployed file is the version that matters.
Prefer useful links over a large list
The proposal encourages links to content that is easy for agents to consume, including clean Markdown versions of pages when they are available. :contentReference[oaicite:1]{index=1}
You do not need to create separate Markdown copies manually just to publish llms.txt.
If your platform already provides maintained Markdown versions, linking to them can make the content easier for agents to retrieve and process. If not, link to the maintained source you actually own rather than creating another copy that may become stale.
The goal is to provide a useful route into the site, not create a second content system.
Test the file as an agent would use it
After publishing llms.txt, test whether it actually points to the information you intended.
Choose a few questions about the site and start from the file.
For example:
- What does the product do?
- How much does it cost?
- How do I get started?
- Which integrations are supported?
- Where is the API documentation?
Check whether the file gives a sensible path to the page that answers each question.
If the answer is missing because the underlying site does not explain it, fix the page first. Adding another link to llms.txt will not repair weak source content.
Frequently asked questions
Does llms.txt make a site appear in AI answers?
No. The proposal does not define a ranking or citation benefit, and publishing the file does not require an AI system to use it.
Does llms.txt replace robots.txt?
No. robots.txt communicates crawler-access rules. llms.txt provides context and links to content that may be useful to agents.
Does llms.txt replace a sitemap?
No. A sitemap is generally a broad list of site URLs for discovery and indexing. llms.txt is intended to be a smaller, curated guide to useful content.
Should llms.txt include every page?
Usually not. Start with the pages that help an agent understand the site or answer important user questions.
Can I have more than one llms.txt file?
Yes. The proposal allows llms.txt at the root or under a subpath, such as /docs/llms.txt, for content beneath that path. :contentReference[oaicite:2]{index=2}
Start small and keep it current
Choose the pages that matter most, write short descriptions, publish the file, and check it after important site changes.
If maintaining the file becomes another source of stale information, simplify it rather than adding more links.
Maverank for agencies helps agencies inspect AI-readiness findings, track prompts, and report changes across client projects.