llms.txt: the complete guide — and what Google actually says about it

You have probably read that you must "add an llms.txt". You may also have read that Google says it does not use one. Both statements are accurate, and the irritation comes from the fact that almost nobody says who the file is written for. This guide answers three questions in order: what the file is, who reads it, and whether it is worth your afternoon.

What llms.txt actually is

llms.txt is a convention proposed in September 2024 by Jeremy Howard (Answer.AI) and documented at llmstxt.org. The idea: a Markdown file, served as plain text at /llms.txt, that introduces your site to a language model — what the organisation is, which pages are authoritative, where the clean documentation lives. It is a curated table of contents, not a permissions file.

The most common mistake is to treat it like robots.txt. robots.txt states what is off limits and follows a real standard, RFC 9309. llms.txt states what matters, forbids nothing, and is standardised by nobody. A crawler that ignores your llms.txt has broken no rule.

The second mistake is to confuse it with a sitemap. An XML sitemap is exhaustive and written for a machine that indexes: it lists everything, flat, with no hierarchy of meaning. llms.txt is selective and written for a machine that summarises: few links, each with a sentence explaining why it is there. If your llms.txt holds all 800 of your URLs, you have written a sitemap in the wrong format.

The format, line by line

The expected structure has four parts: an H1 title (the name of the entity), a blockquote summarising the site in one or two sentences, H2 sections holding Markdown link lists with a description after the colon, and an optional section — conventionally named Optional — for what can be dropped when context runs short.

# Riad Zitoun

> An eight-room riad in the Marrakech medina, open year-round, family-owned since
> 1998. Direct booking, no intermediary.

## Reference pages

- [Rooms](https://example.ma/rooms): all 8 rooms, floor area, bedding, price by season
- [Breakfast and dinner](https://example.ma/table): times, menus, allergens
- [Finding us](https://example.ma/directions): exact address, taxi drop-off, nearest guarded parking
- [Cancellation terms](https://example.ma/terms): deadlines, refunds, deposits

## Verifiable facts

- Address: 12 derb Zitoun, medina, Marrakech 40000, Morocco
- Phone: +212 5 24 00 00 00
- Rooms: 8 · Sleeps: 19 · Check-in: from 2pm
- Languages spoken: Arabic, French, English, Spanish

## Optional

- [Riad journal](https://example.ma/journal): seasonal posts, useful but not essential

Three details separate a file that gets read from one that gets skipped. URLs must be absolute: the file is often read out of context, where a relative link has no base to resolve against. The description after the colon must be informative — "our services" earns nothing, "price by season and cancellation policy" actually steers the read. And the file must be short: it is read alongside the user's question, inside a shared context window.

What Google says — the uneditorialised version

On 15 May 2026 Google published a guide to optimising for generative AI features in Search. Its position on files of this kind is stated plainly in the guide itself: you do not need to create new machine-readable files, special markup or Markdown to appear in Google Search, including its generative features, because Google Search does not use them. Keeping an llms.txt neither helps nor hurts ranking.

That clarity is good news: it moves the argument off belief and onto audience. The real question was never "does it work" — it is "who reads it".

Who actually reads the file

Three families of reader, very different from one another, with very different stakes.

• Coding agents and developer tooling. Tools that work on code — IDE assistants, documentation agents — reach for llms.txt because it saves them parsing a JavaScript-rendered page. This is today the most established and most measurable use.
• On-demand agents. When a person pastes your URL into an assistant, the fetch happens immediately, under an identifiable User-Agent (ChatGPT-User, Claude-User, Perplexity-User). That traffic is triggered by a human who already has you in mind: the most qualified visit in your log file.
• Third-party GEO tooling. Visibility dashboards, agent-facing search engines, specialist directories. Small individually, but they feed the comparison pages that other models later lean on.

What the list does not contain matters just as much: neither Googlebot nor classic indexing. llms.txt is a file for agents, not for search engines. Treat it that way and it stops being disappointing.

Check who is reading you — three commands

Theory is no substitute for your own logs. On an Apache or nginx server, one line tells you whether an agent has ever asked for the file:

# Who requested llms.txt, and under which User-Agent?
grep 'llms.txt' /var/log/nginx/access.log | awk '{print $1, $12, $13, $14}' | sort | uniq -c

# Which AI agents came by this week, across all URLs?
grep -Ei 'GPTBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Google-Extended|Bytespider' \
  /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

# Does the file really answer 200, as plain text, with no surprise redirect?
curl -sSI -L https://your-domain.com/llms.txt | grep -Ei 'HTTP/|content-type|location'

That third check catches the most common and most invisible failure: the soft 404. Plenty of hosts return the homepage, with a 200 status, for any unknown URL. Your llms.txt therefore "exists" as far as any naive checker is concerned, while it is serving HTML. We fixed this in our own audit after watching it slip through: a missing file that answers 200 is worse than a 404, because it reports itself as healthy.

llms.txt, llms-full.txt, /ai/index.md, sitemap.xml: which does what

One caution about the Markdown twin: if the same URL can answer in HTML or Markdown depending on the Accept header, the server must send Vary: Accept. Without it an intermediate cache can hand Markdown to the next human visitor. We were caught by exactly this on our own domain; the header is now set unconditionally.

The seven mistakes that make the file useless

1. Pasting your sitemap into it. 400 undescribed links tell a model nothing; they burn context and push the useful part out of reach.
2. Relative URLs. The file gets read away from its origin; /rooms resolves to nothing there.
3. Promotional descriptions. "The best riad in Marrakech" is not a usable fact. "8 rooms, open year-round, free cancellation up to 48h" is.
4. Unverifiable claims. A model that cites you is staking its own reliability; it favours what it can corroborate elsewhere on your site.
5. Letting it rot. A file advertising 2024 opening hours produces a wrong citation — worse than no citation at all.
6. Putting it anywhere but the root. It is https://domain.com/llms.txt, not a subfolder.
7. Serving it as HTML. Content-Type must be text/plain. A styled error page answering 200 counts as a missing file.

So — should you write one?

The honest verdict, in two lines. If you expect a Google ranking gain: no, the documentation says the opposite and nobody can promise you otherwise. If your audience includes developers, if your product has documentation, or if you already see ChatGPT-User and Claude-User in your logs: yes, because two kilobytes costs half an hour and the file works for those readers.

The real benefit is often indirect anyway. Writing a good llms.txt forces you to decide which five pages are authoritative, to state what the business does in one sentence, and to list facts that can be checked. Most sites discover at that moment that they cannot answer — and that clarification pays off everywhere else: in the JSON-LD, in the meta descriptions, and on the homepage.