llms.txt: what it is, what it's for, and how to create one

llms.txt is a Markdown file you publish at the root of a website to give language models an index of its most useful pages. It takes ten minutes to write. As of September 22, 2026, none of Google, OpenAI, Anthropic or Perplexity has said it uses the file to decide whom to cite.

That last sentence rarely shows up in the guides that explain how to create one. We published ours on September 3, when our new site went live, and we've been counting who requests it in our server logs ever since. Over nineteen days, the crawlers from OpenAI, Anthropic and Perplexity visited auroraglobalgroup.com every day, more than two thousand requests between them, and not one of those requests went to llms.txt. Below: what the file is, what each company says about it, what those logs show, what it's actually good for, and how to write yours, with a downloadable template and our own file annotated section by section.

What is llms.txt?

llms.txt is a proposed standard with no formal backing from any standards body: a plain Markdown file served at /llms.txt that sums up what a site is about and links to its main pages, each with a one-line note. Jeremy Howard proposed it on September 3, 2024, at llmstxt.org, and version 2 of the spec came out on August 10, 2026. The reasoning is practical. A web page is built for a person with a browser, complete with menus, scripts and ads, while a model reading it works inside a limited context window. The file hands the model the summary and the map in a format it can use without parsing HTML. The proposal adds a second convention: offer each page in Markdown as well, at the same URL ending in .md, so an agent can read the content without the layout.

The spec requires exactly one thing: an H1 with the name of the project or site. Everything else is optional and follows this order: a blockquote (the line starting with >) with a one- or two-sentence summary, any paragraphs of context you need, and H2 sections containing lists of links in the format [name](url): note. A section titled "Optional" flags what an agent can skip when it's short on context. There's also llms-full.txt, a dump of the site's full text in a single file. It isn't part of the llmstxt.org proposal, but the developer docs of OpenAI, Anthropic and Perplexity all publish both files.

How is it different from robots.txt and a sitemap?

robots.txt grants permissions, a sitemap lists inventory, and llms.txt provides context: three files at the root, written for three different readers. robots.txt is a standard (RFC 9309) that tells each crawler what it may read, and the automated crawlers from Google, OpenAI, Anthropic and Perplexity all state in their documentation that they follow it. An XML sitemap lists every indexable URL so a search engine doesn't miss any. llms.txt picks a handful, explains them in plain language, and no search engine has committed to reading it.

The order of priority is clear. If robots.txt blocks AI crawlers, llms.txt does nothing, because the model can't open the pages it points to. The first thing we check in any audit is whether robots.txt lets through the bots that answer questions (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) and whether the server returns them the same page it returns to a person. We cover that in detail in SEO for AI.

Which AI engines read llms.txt today?

None of the major ones has said it uses llms.txt for search or citations, and Google has said in writing that it doesn't. Its official guidance on AI features in Search is explicit: you don't need to create new machine-readable files, AI text files or special markup to appear in AI Overviews or AI Mode. Google's John Mueller went further in June 2026, calling llms.txt "purely speculative for now" because the file has been around for years and none of the AI systems use it. In January he had already answered "no" when asked whether llms.txt files on Google properties amounted to an endorsement. OpenAI, Anthropic and Perplexity haven't commented. Their crawler pages explain how to control their bots through robots.txt and say nowhere that those bots read the llms.txt of the sites they visit.

The irony is that all three companies publish their own llms.txt. We checked today: developers.openai.com/llms.txt, docs.claude.com/llms.txt and docs.perplexity.ai/llms.txt all return 200, and each sits next to an llms-full.txt several megabytes long. The reason is usage: developers load that documentation into coding assistants like Cursor or Claude Code, and a Markdown index saves them work. There's a second signal pointing the same way. Lighthouse, Chrome's auditing tool, now checks whether a site has an llms.txt as part of its agentic browsing audits, and its user agent shows up twice in our logs requesting the file.

How often do AI bots request our llms.txt?

Zero times in nineteen days. Between September 4 and 22, 2026, the auroraglobalgroup.com logs show daily visits from GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot, Claude-SearchBot and Claude-User (Anthropic), and PerplexityBot and Perplexity-User (Perplexity): more than two thousand requests in total, spread across the blog, the service pages and the case studies. None went to /llms.txt or /llms-full.txt. Googlebot didn't ask for it either. The only model crawler that fetched it was CCBot, from Common Crawl, once. CCBot builds the open corpus many models are trained on, and it doesn't answer anyone's questions.

So who did request it? Two in three requests came from scripts (curl and node), many of them our own checks. One in six came from site auditing and profiling tools that check whether the file exists: Semrush, BuiltWith, SEOJuice, Arrivl, GEO-Brain, Lighthouse. The rest came from browsers, meaning people who opened it by hand. Right now the file is read mostly by the people auditing sites for GEO. The commands to run the same count on your own server are in how to tell if ChatGPT cites your website. Just swap the path you're searching for with /llms.txt.

So what is llms.txt good for?

It serves AI agents that read a site on someone's behalf, and there it has a real use. When someone asks an assistant to read your site to prep for a meeting, compare vendors or integrate your API, an agent that finds a clean index does better work than one guessing at the structure from your navigation menu. That's why developer docs publish one, and why it makes more sense the more technical the product: a SaaS with an API, a store with a long catalog, a manufacturer with spec sheets. For a twenty-page corporate site, the measurable effect today is close to zero.

Our position hasn't changed since our complete guide to GEO: we publish one because it's cheap, it does no harm, and it's there for any agent that looks for it, but we don't sell it as a way to show up in ChatGPT. Citations come from other things, which we explain in what GEO is: content with extractable answers, sourced data, open crawler access and authority beyond your own site. When we review a site as part of our GEO service, llms.txt takes up one line of the report. Crawler access and answer structure take up the rest.

How do you create an llms.txt, step by step?

It takes five steps and no special tools: a text editor and access to your domain's root.

1. Pick the pages. Somewhere between ten and forty: services or products, case studies, the guides that best answer your customers' questions, and contact. If you wouldn't link a page in an email to a client, leave it out.
2. Write the title and summary. # Company name, then a line starting with > that says what you do, for whom and from where, with at least one verifiable fact.
3. Group into H2 sections. ## Services, ## Case studies, ## Guides. Each line follows the format - [Title](absolute URL): what they'll find there, in one sentence with specifics.
4. Publish at the root. The file lives at https://yourdomain.com/llms.txt, served as plain text (text/plain or text/markdown) in UTF-8 so accented characters don't break.
5. Check that it can be read. Open it in a browser, confirm it returns 200 and that no link 404s, then check that your llms.txt can be read with the AI visibility checker, which also reviews robots.txt.

Here's the template we use, ready to copy or download:

# Company name

> What the company does, for whom and from where, in one or two sentences with a verifiable fact.

## Services

- [Service 1](https://yourdomain.com/service-1/): what it includes and who it's for.
- [Service 2](https://yourdomain.com/service-2/): what it includes and who it's for.

## Case studies

- [Case study 1](https://yourdomain.com/case-studies/case-1/): industry and result, with a number.

## Guides

- [Guide 1](https://yourdomain.com/blog/guide-1/): the question it answers.

## Optional

- [About](https://yourdomain.com/about/): history, location and team.
- [Contact](https://yourdomain.com/contact/): how to reach you and how fast you reply.

On WordPress, Yoast SEO and Rank Math both generate the file, filling it from your published posts and pages and letting you choose what goes in. If you use one, review the output and trim it: an index only works when someone has chosen what belongs in it. Three mistakes we see often: relative links an agent can't resolve, descriptions copied from the meta description that say nothing, and a file nobody updates after the site changes.

What does a real llms.txt look like? Ours, annotated

A real llms.txt is short, has a fact in every line, and gets updated along with the site. Ours is at auroraglobalgroup.com/llms.txt: about 18 KB, in Spanish with an English section, and it follows the spec to the letter. Section by section:

• # Aurora Global Group. The only required field.
• The > blockquote. What we are (a digital marketing and AI agency), where we are (Barcelona), how long we've been at it (more than fifteen years), what we do and how we measure it. It's the sentence an agent would copy if asked to describe us in one line, so it carries facts and no adjectives.
• ## Servicios (services). The services index plus all eight service pages, each with its specific promise: flat fees for paid media, an initial audit for SEO and GEO, an up-front assessment for AI.
• ## Casos de éxito (case studies). Eleven case studies, each with its headline number right in the line (357 leads, 613 franchise applications at 8 euros each, open rates up from 21.9% to 42%). An agent that reads only the index still walks away with the result.
• ## La agencia (the agency). Home, method, contact, Aurora Health and who we are: the pages that place the company before an agent digs into a service.
• ## Blog and ## Herramientas (tools). The guides that answer real questions, and the AI visibility checker, with what it does and what it doesn't.
• ## Contacto y perfiles (contact and profiles). Email, location and LinkedIn in three unlinked lines, so the agent doesn't have to open the contact page.
• ## English version (US). The site has an English twin; this section mirrors the same block structure (agency, services, tools, case studies, blog) and points to the matching pages, so an agent working in English doesn't have to translate on its own.
• ## Contenido completo (full content). The link to llms-full.txt, with the site's full text in a single file.

The llms.txt gets updated every time we publish a page, in the same step and the same commit as the sitemap.

FAQ

How we handle it at Aurora

If you want to know whether your llms.txt exists, is served correctly and lets the right bots through, the AI visibility checker checks it for free in twenty seconds, along with robots.txt and structured data.

If what you want is for AI to cite you, the file is the last thing we touch. First we measure which questions in your industry you show up for and who appears instead of you; that's the first measurement in our GEO service.