/robots.txt is the exception. Everything else is refused at twice the rate.
680 origins, five client profiles, two unrelated networks, 47,600 requests, all paced at a second or more per host. We set out to measure whether sites gate the documents written for AI agents. They do — but not because those documents are written for AI agents, and only about half of the gap is about being an agent at all.
The headline we drafted was wrong, and the table that refutes it is our own.
The study was written up as the file written for AI agents is the one agents are refused most. On the datacenter arm the declared agent is refused /llms.txt on 30.6% of 680 origins — and /sitemap.xml on 30.7% of 680, which is one origin more. A sitemap is addressed to search crawlers, not agents. Nearly nine in ten of the origins refusing one refuse the other.
So the effect is not about AI. /robots.txt is uniquely permitted — refused on only 15.1% of 680 origins from the datacenter arm — almost certainly because it is a well-known path that CDN and WAF layers special-case. Every other document gets the default treatment.
The two-tier edge mechanism
Tier 1 — the base edge rule. A client presenting a browser’s TLS handshake, claiming nothing about being an agent and asking for the identical files, already meets an elevated refusal rate away from /robots.txt: +6.5 points on the datacenter arm, +5.1 on the consumer one. The path is special-cased for every client type.
Tier 2 — the identity penalty. A declared AI user-agent pays that and a penalty on top, at +14.7 points and +13.7 — roughly double. Our first draft reported only the second number and read all of it as an AI effect. About half of it is not.
What each endpoint actually returned.
Declared agent, datacenter arm, 680 origins. A 404 is an answer, not a refusal — and for /llms.txt it is the ordinary case. Counting those as refusals is the single mistake that would have manufactured this study’s entire finding; section 3 shows the check.
| endpoint | distribution | served | absent | refused |
|---|---|---|---|---|
| /robots.txt | 559 · 82.2% | 15 · 2.2% | 89 · 13.1% | |
| /sitemap.xml | 269 · 39.6% | 199 · 29.3% | 190 · 27.9% | |
| /llms.txt | 130 · 19.1% | 339 · 49.9% | 189 · 27.8% |
| outcome | /robots.txt | /llms.txt | /sitemap.xml |
|---|---|---|---|
| 200 served | 559 · 82.2% | 130 · 19.1% | 269 · 39.6% |
| other served — 2xx/3xx, chiefly 202 | 3 · 0.4% | 3 · 0.4% | 3 · 0.4% |
| 404 absent — an answer | 15 · 2.2% | 339 · 49.9% | 199 · 29.3% |
| 403 refused | 75 · 11.0% | 150 · 22.1% | 153 · 22.5% |
| 402 refused | 2 · 0.3% | 17 · 2.5% | 11 · 1.6% |
| 429 refused | 3 · 0.4% | 7 · 1.0% | 11 · 1.6% |
| other refused — 406, 451, 500, 503 | 9 · 1.3% | 15 · 2.2% | 15 · 2.2% |
| no response — transport | 14 · 2.1% | 19 · 2.8% | 19 · 2.8% |
The rule is the path, not the address
If datacenter IP reputation drove this, the AWS arm would be the worse of the two. It is the better one on every endpoint, and the gap survives with the network held constant inside each arm.
| endpoint | datacenter · AS16509 | consumer · AS5483 | gap vs /robots.txt |
|---|---|---|---|
| /robots.txt | 13.1% | 15.6% | — |
| /llms.txt | 27.8% | 29.3% | +14.7 / +13.7 |
| /sitemap.xml | 27.9% | 29.0% | +14.9 / +13.4 |
What declaring an identity costs, path by path
The gap column above reads across a row — one client, two paths. Reading down instead gives the other effect: same path, same handshake, and the only difference is that one client says what it is.
| path | client | refusal | refused | identity penalty |
|---|---|---|---|---|
| /robots.txt | browser TLS handshake | 24 · 3.5% | ||
| declared AI agent | 89 · 13.1% | +9.6 pts | ||
| /llms.txt | browser TLS handshake | 68 · 10.0% | ||
| declared AI agent | 189 · 27.8% | +17.8 pts |
And the same origins do it on both networks. 81 origins from the datacenter arm and 73 from the consumer arm serve /robots.txt and refuse both other endpoints, and 71 of them are the same origins — a Jaccard index of 0.86 over the two sets. Address reputation would not select the same sites from two unrelated networks. On most of them the same server header issues the 200 and the refusal, so it is one layer applying a path rule rather than two systems disagreeing.
Three origins serve a declared agent a different policy than they serve a browser.
This result is not from the AIR study. It comes from the measurement of 26 August, which this study supersedes and which is not itself published here, and it is reproduced in full because it is that study’s sharpest finding and because the disclosure notes name this page as where the affected publishers are identified. Retiring a study must not retire a result.
The funnel is the finding’s own control. Sixteen per cent of comparable origins differ at the byte level — timestamps, cache keys, generated comments — and almost all of it evaporates by the level that maps to behaviour. Reporting the byte-level number as “the policy differs by client” would have inflated this result twentyfold.
Three origins are established as a delivery-layer effect: www.forbes.com, www.rollingstone.com and www.blic.rs. Each served the declared agent a 597-byte file that is byte-identical across five unrelated publishers in four countries, names none of them, and carries no Sitemap: directive — while every other client received the site’s own, entirely different file, within the same seconds. A static file cannot do that.
What layer answers the agent’s request is not established, and this page names no such layer. Nothing here attributes the behaviour, or any intent, to the publishers. The simpler reading is that the agent’s request is answered by a different service than the one answering browsers, and that service served its own default.
The full chain of evidence, the header analysis behind it and the two origins removed from an earlier count of five are in reports/cp-pub/attribution.md.
Education publishes least and refuses most.
The corpus is stratified into eight classes, and the path rule does not fall evenly across them. Adoption of /llms.txt spans a factor of five, and the sector most people expect to be strictest — news, with its licensing deals and its WAFs — sits in the middle of the refusal range rather than at the top of it.
| stratum | origins | outcome | served | refused |
|---|---|---|---|---|
| SaaS & B2B | n=63 | 23 · 36.5% | 12 · 19.0% | |
| E-commerce | n=108 | 35 · 32.4% | 36 · 33.3% | |
| Corporate | n=31 | 7 · 22.6% | 3 · 9.7% | |
| Community | n=101 | 18 · 17.8% | 32 · 31.7% | |
| Reference | n=79 | 14 · 17.7% | 13 · 16.5% | |
| News | n=180 | 24 · 13.3% | 56 · 31.1% | |
| Government | n=57 | 5 · 8.8% | 15 · 26.3% | |
| Education | n=61 | 4 · 6.6% | 22 · 36.1% |
Publishing and refusing are not opposites here. E-commerce is second in adoption and second in refusal at the same time: the same stratum that writes the file is the one most likely to have an edge rule that will not hand it over. What separates the strata is not a policy about AI, it is how much machinery sits in front of the origin.
Two defects that would each have produced this finding out of nothing.
404 disambiguation — an absent file is an answer
/llms.txt genuinely does not exist at 339 of 680 origins on the datacenter arm. It is a new convention and most sites have not adopted it. Had those 404s been classified as refusals, the endpoint’s refusal rate would read close to three times what we publish, and the whole finding would be an artefact of an optional file being absent rather than evidence of anything.
Zero of those 339 are counted as refusals — verified across all three endpoints on both networks, and asserted in the generated report rather than trusted to a code comment. It is the single decision this study most depends on, so it fails loudly if it ever regresses.
The parser read challenge walls as permission
A separate measure asks how often the edge refuses a URL that the site’s own robots.txt permits. It was first published at roughly twice the rate reported below, and the difference was entirely this defect.
84 origins answered the agent’s request for /robots.txt with an HTTP 403 challenge page, and the harness stored that HTML as though it were the policy. A robots parser reads a page of markup as a ruleset containing no rules — which means everything is permitted. The origin’s own refusal then became evidence that it contradicts its own published policy.
Corrected to count only origins that actually served the file, the rate is 8.9% of 1,902 agent page requests on the datacenter arm and 9.4% of 1,732 on the consumer arm. An origin that refuses you its policy has published no rule you could have obeyed, and counting it as a contradiction was the same error in a different coat.