KhaqanWhat is llms.txt?
llms.txt is a plain Markdown file placed at a website's root (/llms.txt) that gives AI assistants a short summary of the site and a curated list of its most useful pages, so a language model can find the right content quickly.
The llms.txt proposal was published by Jeremy Howard of Answer.AI in September 2024. It asks site owners to put a Markdown file at /llms.txt with a fixed shape: an H1 with the site or project name, a blockquote with a one-sentence summary, optional paragraphs of context, and then H2 sections that each hold a list of links written as a dash, the title in square brackets, the URL in round brackets, and a short description after a colon. A final section headed Optional marks links that can be skipped when an assistant is short of context.
The idea matters because most business websites are built for browsers, not for models. Navigation menus, cookie banners and scripts get in the way, and a model with a limited context window cannot read a whole site. A well-written llms.txt is a table of contents for machines: it points to the pages that answer real questions, such as services, pricing, locations and contact details.
Adoption is uneven and worth stating plainly. Documentation-heavy companies publish one, and a number of documentation tools generate it automatically, but none of the major AI companies has confirmed that its crawlers or assistants use llms.txt, and Google's search advocates have said publicly that Google does not. I publish one on every site I run because it takes ten minutes, it doubles as a content audit, and the audit tool I wrote, ai-visibility-audit, checks that it exists and that its links resolve.
The common mistake is treating it as a sitemap and listing every URL. A good llms.txt is short, curated and honest about what each page contains; a bad one is a dump of links with no descriptions.
Related terms
- llms-full.txt: llms-full.txt is a companion to llms.txt that holds the complete text of a site's key pages in one Markdown file, so an AI assistant can load everything it needs in a single request instead of following links page by page.
- GEO (generative engine optimisation): Generative engine optimisation (GEO) is the practice of making a website easy for generative AI systems such as ChatGPT, Perplexity, Claude and Gemini to crawl, understand and cite, so that the site appears as a source in AI-generated answers.
- AI crawler (GPTBot, ClaudeBot, PerplexityBot and others): An AI crawler is a bot that fetches web pages for an AI vendor to train models, to build a search index for an assistant, or to fetch a page live when a user asks; most vendors use a separate user agent for each purpose.
- AI search visibility: AI search visibility is how often, and how accurately, a website is used and cited by AI-driven search products such as ChatGPT search, Perplexity, Google AI Overviews, Copilot and Claude when people ask questions the site could answer.