Skip to content
siterank.info
Sign in Start free

AI crawlers

What Is llms.txt? What It Can and Cannot Do

llms.txt is a community proposal for a Markdown site map aimed at LLMs. Here is the format, its real status, and what publishing one does not do.

By ankitkumarvig@gmail.com Published September 9, 2026 Last reviewed September 9, 2026 Sources verified September 9, 2026 12 min read

llms.txt is a proposed convention for a small Markdown file at the root of a website that tells a language model or an agent where the site's most useful content lives and how to read it. It was proposed by Jeremy Howard in September 2024, has been revised since, and is now published by thousands of sites. It is also the subject of more confident claims than the evidence supports. This article describes the format as currently specified, what is known about who reads it, and how to publish one from WordPress without misrepresenting what it will achieve.

The proposal in one paragraph

The rationale on llmstxt.org is that web pages are built for people: navigation, advertising, scripts and layout wrap the content, and turning HTML back into clean text is imprecise and expensive inside a limited context window. llms.txt offers a curated, expert-written summary of a site plus a short list of links to the pages that matter, so that a model or agent can orient itself cheaply. It is a map, not the territory: the file points to content rather than replacing it.

Two related conventions travel with it. The proposal suggests that pages also be made available as Markdown, either at the same path with a .md extension or with .html.md appended. And it suggests link relations (rel="alternate" type="text/markdown" and rel="describedby"), usable as HTML elements or HTTP headers, to connect a page to its Markdown version and to the site's llms.txt.

Status: a community proposal, not a standard

This is the part that matters most, so it comes first.

  • llms.txt is not an RFC, not a W3C recommendation, and not part of the Robots Exclusion Protocol. It is a proposal maintained by its author with community input.
  • No search engine documents it as an input. Google's documentation on AI features and your website states that there are no additional requirements to appear in AI Overviews or AI Mode, and specifically that you do not need to create new machine-readable files, AI text files or markup to appear in those features.
  • Google's John Mueller wrote on Bluesky in June 2025 that, as far as he was aware, no AI system currently used llms.txt, and pointed to server logs as the evidence: chatbots fetch pages for training and grounding, but they do not fetch the file. That is a personal statement by a Google employee rather than formal documentation, but it is consistent with the documentation above and we have found no provider documentation that contradicts it.
  • OpenAI, Anthropic and Perplexity each document their crawlers in detail (see AI crawlers explained). None of those crawler pages mentions llms.txt.

The proposal site itself reports genuine adoption: thousands of published files, documentation platforms that generate one automatically, a Lighthouse audit that checks for one, and vendor files published by OpenAI, Anthropic and Google. All of that is evidence that people publish llms.txt. It is not evidence that any particular model consumes it. The distinction is the whole subject: presence is not ingestion, ingestion is not retrieval, and retrieval is not citation.

On the SiteRank evidence tiers, publishing llms.txt is EMERGING PRACTICE: widely adopted, with a plausible mechanism, and not confirmed by any provider as an input to retrieval or answer generation.

The specified format

The current text of the proposal specifies a Markdown file at /llms.txt, which may also be placed under a subpath such as /docs/llms.txt to describe URLs beneath that path. The structure, in order, is:

  1. An optional byte-order mark.
  2. An H1 with the name of the project or site. This is the only required element.
  3. A blockquote containing a short summary with the key information needed to understand the rest of the file.
  4. Zero or more Markdown sections of any type except headings (paragraphs, lists) giving more detail about the project and how to interpret the linked files.
  5. Zero or more sections delimited by H2 headings, each containing a "file list": a Markdown list whose items are a required hyperlink [name](url), optionally followed by a colon and notes about that file.

One H2 heading has reserved meaning. A section titled Optional holds secondary links that an agent may skip when it needs a shorter context.

A minimal valid file for a WordPress site might look like this:

# Example Bakery

> Example Bakery is an independent sourdough bakery in Leeds. This site
> documents our breads, ordering process, wholesale terms and allergen
> information.

## Products

- [Sourdough range](https://example.com/breads/): every loaf we bake, with ingredients and allergens
- [Wholesale](https://example.com/wholesale/): minimum orders, delivery days and pricing structure

## Policies

- [Allergen statement](https://example.com/allergens/): how we handle nuts, gluten and dairy
- [Contact](https://example.com/contact/)

## Optional

- [Press coverage](https://example.com/press/)

Notice what is not in the format: there are no directives, no Allow or Disallow, no user-agent groups, no rate hints, and no way to say "use this" or "do not use that". The file is descriptive. It can guide a reader that chooses to read it; it cannot control one.

What about llms-full.txt?

llms-full.txt is a separate community convention for a single file containing the full text of a site's important content, rather than links to it. It is not part of the proposal document at llmstxt.org and there is no specification for it. Treat it as EXPERIMENTAL. It also has a practical hazard: for a site of any size it becomes megabytes of text that must be regenerated whenever content changes, and an unbounded generator is a way to make your own server do expensive work on demand for anyone who requests the URL.

What llms.txt is not

Because the file's name resembles robots.txt, it is often described as if it belonged to the same family. It does not.

File Governs Who reads it Standing
robots.txt Whether a compliant crawler may fetch a URL Every major search and AI crawler, by documented policy ESTABLISHED STANDARD (RFC 9309)
XML sitemap An inventory of URLs for crawlers to discover Search engines, by documented policy OFFICIAL PROVIDER GUIDANCE (sitemaps protocol, supported by Google and Bing)
llms.txt A curated, human-written orientation for a model or agent No provider documents reading it EMERGING PRACTICE

robots.txt controls access; llms.txt offers orientation. Neither substitutes for the other. Publishing llms.txt does not grant or deny any crawler anything, and it never justifies neglecting a sitemap.

Use cases where it plausibly helps

The proposal's own worked examples are documentation sites, and that is where the case is strongest. The mechanism is not "an AI provider will read this file" but "a person or an agent that has been pointed at this file gets a better starting point than the HTML".

  • Developer documentation. A user pastes https://example.com/docs/llms.txt into a coding assistant, or a tool ingests it deliberately. The file gives the assistant a map of the reference pages in a few hundred tokens.
  • Product and reference sites where a small number of canonical pages answer most questions: pricing, specifications, policies, an about page.
  • Agent workflows you control. If you build or configure an agent that operates on your own site, you decide what it reads, and a curated map is cheaper than crawling.
  • Editorial clarity. Writing the file forces a decision about which ten or twenty pages actually define the site. That exercise has value even if the file is never fetched, and it tends to surface the same pages that topical authority analysis identifies as pillars.

None of these is "appear in ChatGPT". If the person or tool consuming your site never requests the file, the file does nothing.

Limitations and what not to assume

  • Do not assume a model reads it. No provider documents doing so. If you want to know whether anyone fetches yours, check your server logs for requests to /llms.txt; that is the only evidence that exists.
  • Do not present it as an SEO or GEO requirement. Google says the opposite. An audit that flags "missing llms.txt" as a defect is flagging something that no documented system needs.
  • Do not expect it to change visibility. Attributing a change in AI mentions to publishing the file would require a controlled before-and-after measurement, and even then confounders make causal attribution very weak. See how LLM visibility monitoring works for what an honest measurement involves.
  • Do not use it to route around noindex or Disallow. A URL your own robots policy disallows, or a page marked noindex, has no business in an orientation file. Listing it is contradictory at best.
  • Do not list everything. A file that enumerates every post on the site is a worse sitemap, not a better guide. The value is selection.
  • Do not write marketing copy into the descriptions. The notes after each link are for a reader that wants to know what the page contains, not for persuasion. Keyword-stuffed descriptions are the same mistake as keyword-stuffed meta descriptions.
  • Do not let it shadow a sitemap. Search engines document sitemaps; they do not document llms.txt.

The behaviour of the file, if it is read at all, is also a matter of the reader's choice. There is no way to compel a reader to prefer it, and no way to know from the file alone whether the reader treated it as authoritative.

Implementing it on WordPress

WordPress does not ship an llms.txt feature. The proposal site lists several SEO and hosting products that generate one, so if you already run one of those, check what it produces before adding another; two plugins fighting over the same URL is a common source of stale or duplicate output.

There are two ways to serve the file, and the difference matters.

A physical file at the document root is simple and fast, but it is invisible to WordPress. Nothing regenerates it when content changes, deployment tooling may overwrite it, and it silently overrides any plugin that expected to serve the URL itself.

A virtual endpoint is served by WordPress through a rewrite rule, so the output can be regenerated from current content, cached, and reviewed before it changes. That is how WordPress itself serves robots.txt when no physical file exists.

SiteRank AI takes the second approach, with a few constraints that follow from the limitations above:

  • It drafts a candidate file from the site's own content, proposing pages that the local topic clustering identifies as pillar candidates, plus reference pages such as about, contact and policies. Every entry can be included, excluded, reordered or rewritten. Your edits are stored separately from the generated candidates, so regenerating after new content never silently discards a hand-written description.
  • It validates the draft against the proposal's format: exactly one H1, a blockquote summary, H2 sections that contain link lists, absolute URLs that resolve, and no URL that the site marks noindex or that its own robots policy disallows. Failures are reported as file-correctness issues, never as visibility issues.
  • It previews the exact bytes that will be served.
  • It publishes only on an explicit action, as a virtual /llms.txt served as text/plain with a cache header. Nothing is written to disk. If a physical /llms.txt already exists at the document root, the plugin reports that the virtual endpoint is shadowed rather than competing with the file.
  • It never scores the file's existence. There is no "your site is missing llms.txt" warning, because absence is not a defect under any documented system.

If you prefer a physical file, write the validated output to disk yourself; the trade-off is that you own its freshness from then on.

Evidence classification

Practice Tier Note
Publishing a curated llms.txt EMERGING PRACTICE Widely adopted; no provider documents consuming it
Publishing llms-full.txt EXPERIMENTAL Not specified; size and cost hazards
Serving Markdown variants of pages EMERGING PRACTICE Suggested by the proposal; no provider documents requesting them
Treating absence of llms.txt as an SEO or GEO defect Contradicted by OFFICIAL PROVIDER GUIDANCE Google states no AI text files are needed
Expecting llms.txt to produce citations HYPOTHESIS No mechanism documented; crawled is not cited

Key takeaways

  • llms.txt is a Markdown orientation file proposed by Jeremy Howard in 2024. The only required element is an H1; a blockquote summary and H2 link lists are optional.
  • It is a community proposal, not a standard, and no search engine or AI provider documents reading it. Google states that no AI text files are needed to appear in its AI features.
  • It is unrelated to robots.txt: one describes, the other controls access. It does not replace a sitemap.
  • Its plausible value is for readers you point at it, such as coding assistants and agents on documentation sites, and as an editorial exercise in deciding which pages define the site.
  • Publish it curated, validated, and on purpose. Do not expect it to move any visibility number, and do not treat its absence as a problem.

Official sources & further reading

Frequently asked questions

Does publishing llms.txt actually do anything today?

For most sites, nothing you can observe. No search engine or AI provider documents reading the file, so the honest position is that its effect on any provider's behaviour is unknown rather than demonstrated to be small. Where it does help is with readers you point at it yourself — a coding assistant handed the URL, or an agent you configured — and with the editorial discipline of deciding which pages define the site. Publish it for those reasons, not in the expectation of a visibility change.

How can I tell whether anything is fetching mine?

Look in your server access logs for requests to /llms.txt; that is the only direct evidence available to you. Be careful about what a log line proves: it records that something requested the file, not that any model read it, used it, or went on to retrieve the pages it names. Presence is not ingestion, and a fetch is not a citation. If nothing ever requests the file, that is also information worth having.

Should I publish llms-full.txt as well?

Probably not, unless a specific reader is asking for it. It sits at EXPERIMENTAL: there is no specification for it, it is not part of the proposal document, and for a site of any size it becomes a large file that has to be regenerated whenever content changes. If you do publish one, generate it as a bounded artefact on a schedule rather than on demand, so a single request cannot make your server rebuild the whole corpus.

Does llms.txt let me control what AI systems do with my content?

No. The format contains no directives at all — no user-agent groups, no allow or disallow, no rate hints — so there is nothing in it that grants or withholds anything. Access is governed by robots.txt, and even that is voluntary compliance by the operator. Keep the two apart in your head: one describes, the other asks.

How often should I update it?

Whenever the pages it names change materially — a renamed URL, a retired page, a new reference page that belongs in the list. A virtual endpoint regenerated from current content handles most of that for you, whereas a physical file is only as fresh as the last time someone remembered it. Since the file's value is selection rather than coverage, resist the urge to append every new post; review the list occasionally instead, and keep hand-written descriptions separate from generated candidates so a regeneration never quietly discards them.