What is llms.txt?

llms.txt is a plain-text file, conventionally served from a site's root at /llms.txt, modeled deliberately on the decades-old robots.txt convention but aimed at a different audience: large language models and the AI crawlers that feed them, rather than search-indexing bots. Where robots.txt tells a crawler what it's allowed to fetch, llms.txt is meant to tell an AI system what's worth fetching, a curated, Markdown-formatted map of a site's most important pages, written in plain language a model can parse quickly without having to render a full HTML page or guess at site structure.

The proposal's rationale is that language models work under tight context and time budgets, and a typical website's HTML is full of navigation, scripts, and layout markup that has nothing to do with the actual content. A llms.txt file strips that away and hands the model a clean summary: what the business does, links to its most important pages, and short descriptions of what each one covers. That is the intent of the format. Whether any given AI system reads the file, or does anything different because of it, is a separate question.

It's important to be precise about what llms.txt is not. Google has said llms.txt is not needed for Google Search and has no effect on visibility or rankings (Google, June 2026). It may still be used by other AI systems, but support there is unconfirmed: which crawlers read it at all is not established, and there is no evidence that serving one makes a site more likely to be cited. It is also no substitute for page content that is well-structured and extractable, which is what any system has to work from once it fetches a page.

In the Ritam Labs paid platform, llms.txt presence and quality feeds into the AI Crawler Visibility dimension (D5) alongside robots.txt rules and bot-access checks, and the matching fix generator can produce a ready-to-deploy llms.txt alongside a corrected robots.txt, so both the access layer and the summary layer for AI crawlers get addressed together. In the free audit it is one of ten AI Search Readiness checks, and only whether the file exists is tested.