Where should the llms.txt file live on my site?

At the root, as /llms.txt, if you publish one at all. Version 2 of the spec also allows it at a subpath, so /docs/llms.txt covers everything under /docs/, and agents should use the most specific file that applies. Chrome's Lighthouse only requests the root path, so a subpath file is invisible to that audit even when it is correct.

Can an llms.txt file hurt my search rankings?

Not in Google Search. The note Google added to its AI guidance in June 2026 says keeping the file neither helps nor harms visibility, because Search ignores it. The damage is elsewhere. A malformed file scores zero in Lighthouse's Agentic Browsing category, and a stale one hands any agent that does read it a confident description of a site you no longer have.

Is llms.txt the same thing as robots.txt?

No, and the difference is why one works. Robots.txt restricts crawlers, and because restriction is costly to the site, it is credible. An llms.txt file advertises your own content in your own words, so every site can claim to be the best answer. Lighthouse audits both, with the same scoring logic: missing is not applicable, broken is zero.

Do I need markdown versions of my pages as well?

The spec recommends it, and that is the heavier half of the work. It asks for a clean markdown copy of each page at the same URL with .md appended, linked with rel="alternate" type="text/markdown", plus rel="describedby" pointing back to the llms.txt that covers it. Both can be sent as an HTTP Link header from your CDN rather than edited into templates.

Does my website need an llms.txt file?

Say you run a nine-person letting agency in Birmingham. The new site went live in March. It loads in under two seconds, the valuation form drops straight into your CRM, viewings are up. Then a consultant emails you one line: add an llms.txt file so AI assistants can find you. An hour of work, a small invoice, no explanation.

So you forward it to your developer with four words of your own. Does my website need an llms.txt file?

Most business websites don't need an llms.txt file. Google has said its Search features ignore the file, so publishing one won't win you a mention in AI Overviews or AI Mode. The catch is narrower and stranger than the sales pitch: Chrome's Lighthouse now audits llms.txt, and an absent file is scored "not applicable" while a file that exists and is malformed scores zero.

What is an llms.txt file?

It's a markdown file, usually at the root of your domain, listing the pages on your site that an AI agent should read. Jeremy Howard proposed it in September 2024, and published a second version of the spec in August 2026.

The format is tiny. One H1 with the site's name is the only required part. After that: an optional blockquote summary, a few plain paragraphs, then H2 sections holding markdown link lists. Each link should point at something a model can read without wading through your navigation.

Version 2 added the half that everyone skips. It asks you to serve a clean markdown copy of each page at the same URL with .md appended, pointed at by rel="alternate" type="text/markdown", with rel="describedby" pointing back at the llms.txt that covers it. Both can travel in an HTTP Link: header, so a CDN rule does it without touching a template.

That is a build job, not an hour.

Look at what the spec is asking for and you can see who it was built for. The spec says it plainly: these files "are used most heavily for software documentation", where coding agents follow them to find API references and tutorials. A letting agency is not that.

Does Google use llms.txt for AI Overviews or AI Mode?

No. Search Engine Journal reports that Google added a note to its AI guidance in June 2026 saying the file isn't needed to appear in Search, and that keeping one neither helps nor hurts, because Search ignores it.

John Mueller made the structural argument on Google's own Search Off the Record podcast. The same write-up reports him saying the files can't help language model systems decide which site to surface, because the content is self-reported by the site that wants picking. The quote it carries: "It's basically you're telling these systems, like, I have the best website ever."

A field that every competitor can set to maximum carries no information. Robots.txt works because it restricts you, and a restriction is costly, so it's credible. An llms.txt file only ever flatters you.

What does the Lighthouse llms.txt audit actually check?

Three things: an H1, at least one markdown link, and at least 50 characters of content. Miss any one of the three and the check scores zero.

The audit ships inside Chrome's Lighthouse, in a category called Agentic Browsing that landed in Lighthouse 13.2.0 on 30 April 2026, and became available through the PageSpeed Insights API that July. Its own description in the source calls the category "still under development and subject to change", which is unusually candid. Seven checks sit in it: an agent accessibility tree audit, three WebMCP audits, Cumulative Layout Shift, the llms.txt check, and an ARD manifest check. There's no score out of 100. You get a pass ratio.

The llms.txt check itself is about fifty lines of logic you can read in the Lighthouse source. Lighthouse requests /llms.txt at your root, and then:

A 404, or any other 4xx, scores 1 and is marked not applicable. No file, no penalty.
A 500, or a 429 from a rate limiter, scores zero.
A file that loads but has no H1, or no markdown link, or fewer than 50 characters, scores zero.

So every failure mode in that audit belongs to someone who published a file. A plugin that writes an empty stub on activation. Text pasted in with the heading stripped. Your web application firewall (WAF) answering Lighthouse with a 429, because it decided the request was a bot. All zeros, all self-inflicted, none of them possible if you'd done nothing.

A single highlighted path traced through a field of scattered nodes, labelled fetch, check and score, standing for one Lighthouse request for llms.txt.

Cumulative Layout Shift, the Core Web Vital that measures how much a page jumps around as it loads, is in here because an agent clicking a button that moves 200 milliseconds later has the same problem a person does. We've written before about why a site can fail Core Web Vitals on a perfect PageSpeed score, and that work now counts in two places. The agent accessibility tree audit behaves the same way: it rewards semantic markup, real labels, buttons that are buttons, the same things WCAG 2.2 (the current version of the Web Content Accessibility Guidelines) already asks for, and exactly the work an accessibility overlay widget skips. An agent reads your page through roughly the same tree a screen reader does.

Six of the seven checks reward building the site properly. One rewards a text file.

So which websites need an llms.txt file?

The ones with more content than a person can skim: documentation sets, knowledge bases, long reference libraries. Our rule is one question. Can you name the pages the file would point to, and the person who will update it when those pages change?

Worth the hour:

Documentation, a developer reference, a knowledge base, a help centre.
A long library of guides whose structure isn't obvious from the navigation.
A product whose customers paste your docs into Claude or Cursor to get an integration right.

Not worth it:

A six-page service site, where everything worth reading is two clicks from the homepage and the whole site fits in a context window twice over.
A fit-out contractor in Al Quoz with eight pages and a project gallery. Listing those eight pages in a text file tells a model nothing the sitemap didn't.
Any site where nobody has write access to the file except the agency that built it.

The stale file is the real cost. Your llms.txt is a set of claims about your content. Six months later the pricing page has moved, two guides are gone, the service names changed, and the file still describes the old site with total confidence. Nobody is at fault, because updating it was never in anyone's job description. This is the same failure we see when an assistant hands someone a snippet and they paste it into production without owning it, which is its own sign that it's time to bring in a developer.

What we got wrong about llms.txt

We ship an llms.txt file on sites we build, including our own, and we can't show you a single citation it has earned. Not one. For a stretch last year we were charging for it on website development quotes, under AI readiness, and we've taken it off the quote, because we couldn't point at anything in a client's analytics that moved after it went up.

We still generate one, for two honest reasons. It costs us nothing once it's automated, and if an agent is already crawling a site, a correct index is better than a wrong one. Those are weak reasons. They're not the reasons it gets sold.

If you're paying a retainer that includes AI search optimisation and the deliverable is a text file, ask what changed after it shipped. If the answer is a screenshot of the file, you've bought the file.

WebMCP and ARD matter more than llms.txt now

Three of the seven Agentic Browsing checks are about WebMCP, which tells you where Google thinks this is going. WebMCP carries the Model Context Protocol's idea of tools into the page itself: a site registers JavaScript functions as tools, each with a plain-language description and a schema, through document.modelContext.registerTool(). The difference from llms.txt is the difference between describing your booking form and letting an agent submit it.

It's early, but it isn't speculative. The spec's own implementation status records an origin trial live in Chrome 149 and in Edge 150, support in ChatGPT Desktop, and experimental support in Brave's Leo. Firefox has a standards position issue open and nothing more. Origin trials expire, so this one either graduates or lapses inside a year. That's the date to watch.

The quieter one is ARD, Agentic Resource Discovery, a manifest at /.well-known/ard.json that lists the callable resources a domain offers. Lighthouse 13.6.0, released on 27 October 2026, moved its ARD check to version 0.91 of the spec, where ard.json replaces the older ai-catalog.json filename, and now warns you when your manifest is only reachable at the legacy location. Its scoring repeats the llms.txt pattern exactly. No manifest and nothing advertising one is not applicable. Advertise a manifest that 404s and you score zero.

The pattern is worth saying out loud, because it will hold for whatever file comes next: silence is free, and a broken promise is a zero.

What to do about llms.txt this week

Open yourdomain.com/llms.txt. Plenty of people reading this will find something, put up by a plugin or an agency and never looked at since.

If it loads and it's empty, missing its heading, or six months out of date, fix it or delete it. A 404 scores better than a broken file, in Lighthouse and with any agent that reads it. If you want the number, run Lighthouse from the CLI with --only-categories=agentic-browsing and read the pass ratio.

Then spend the hour you saved on the things both standards assume you did first: CLS, a real submit button, a stable URL, labels that say what each field is. That work compounds. A text file doesn't.

If you'd rather someone just told you whether your site is in decent shape for this, send us the URL through our contact page and we'll run the audit and tell you what we'd actually change. If the answer is nothing, we'll say nothing.