Treating llms.txt as a Distribution Plan
Shipping the machine-readable layer is cheap and worth doing. Expecting it to bring readers is a different claim, and the evidence is against it.
A site publishes llms.txt, a markdown mirror and a JSON index, and calls that its growth strategy. The files are worth shipping — they cost hours and they serve the real case where a person hands your link to their agent. What they do not do is bring anyone. Google says no AI service fetches llms.txt; one crawler logs roughly 24,000 crawled pages per single referral. Build the layer, then go find humans.
Pattern
You are building something agent-first, so you ship the machine layer properly: llms.txt, a .md mirror of every page, a JSON catalogue, a permissive robots.txt. It takes an afternoon.
Then the plan quietly becomes: models will read this, models will cite us, readers will follow. The machine layer stops being a feature and becomes the strategy. Nobody writes that down, which is why nobody checks it.
Why it looks right
Because most of the individual beliefs are true, and the layer is genuinely good work.
Machines really are the majority reader now — automated requests crossed 57.5% of HTML traffic in mid-2026. Blocking AI crawlers really does cost you: publishers who blocked lost 7% of weekly traffic in 6 weeks with no measurable protection in exchange.
Both are reasons to build the layer. Neither is evidence that the layer brings readers. The step from "machines read the web" to "machines will send me people" is the one that never gets examined.
Why it fails
Three independent measurements point the same way.
Nobody fetches the file. Google's position, stated plainly:
None of the AI services have said they're using LLMs.TXT, and you can tell when you look at your server logs that they don't even check for it.
An independent scan of 300,000 domains found about 10% had adopted llms.txt with no measurable lift in citations. Adoption is not consumption.
Crawling is not traffic. One AI crawler logs on the order of 24,000 crawled pages per single referral. Roughly half of AI crawl requests serve training, under a tenth serve live search. A crawler arriving is not a reader arriving, and a dashboard that counts bot hits as interest will feel wonderful and mean nothing.
The timescale is wrong for a new site. Practitioners put the first meaningful result from generative-engine work at around 16 months — fine as a background investment, useless as a plan for your first 90 days.
What does correlate with being cited is off-site. In one study of 75,000 brands, mentions on video and in ordinary web text outranked backlinks by two to three times as predictors of AI visibility, and one large forum accounted for 40% of citations sampled. The machine layer is on your server. The signal is not.
Instead
Ship the layer. Demote the expectation. Concretely:
- Keep
llms.txt, keep the.mdmirror, keeprobots.txtopen to AI crawlers. The cost is hours and blocking is measurably worse than allowing. - Serve the mirror as
text/markdown, and put the date and version in the body text, not only in metadata. Cited content skews fresher, and freshness has to survive being pulled out of context. - Move the machine layer out of your success metrics. Percentage of agent traffic becomes a diagnostic you look at, not a number you steer by.
- Do not point anyone at one giant file as the recommended fetch. Longer context measurably degrades extraction accuracy. Give a catalogue and let the agent pick.
- Then go where people already talk — the part the afternoon of file-writing lets you feel you have already done.
The machine layer is a product feature serving the moment someone hands your link to their agent. It is a good feature. It is not a plan.