llms.txt was proposed in 2024 as a companion to robots.txt and sitemap.xml, aimed specifically at LLM training and retrieval crawlers. It is a markdown-formatted file that summarises the site and provides curated links to primary sources.
Contents
A typical file includes an H1 site name, a short site description, and one or more sections of linked resources. Some sites also publish an expanded llms-full.txt that includes body text so LLMs can index without additional fetches.
Example
A developer-tools company publishes /llms.txt with links to its docs, changelog, and API reference. Perplexity and ChatGPT consume the file and cite its docs three times more often on developer queries.
How SEM Optimiser reports it
The AEO audit validates llms.txt against the current spec, checks for broken links, and monitors how often the file is fetched by known LLM user-agents.
Related terms
- AEO (Answer Engine Optimisation)
Answer Engine Optimisation is the practice of structuring content so it is selected and cited by answer-first engines including ChatGPT, Perplexity, Gemini, and Google's AI Overviews.
- GEO (Generative Engine Optimisation)
Generative Engine Optimisation is a broader discipline than AEO, covering how content, brand entity, and technical signals influence generation by large language models across all AI surfaces.
- robots.txt
robots.txt is a plain-text file at the root of a domain that instructs crawlers which paths they are allowed or disallowed to fetch.
- MCP (Model Context Protocol)
The Model Context Protocol is an open standard developed by Anthropic for connecting LLM applications to external data sources and tools through a shared client-server interface.