/robots.txt is the exception. Everything else is refused at twice the rate.
680 origins, five client profiles, two unrelated networks, 47,600 requests, all paced at a second or more per host. We set out to measure whether sites gate the documents written for AI agents. They do — but not because those documents are written for AI agents, and only about half of the gap is about being an agent at all.
The headline we drafted was wrong, and the table that refutes it is our own.
The study was written up as the file written for AI agents is the one agents are refused most. On the datacenter arm the declared agent is refused /llms.txt on 30.6% of 680 origins — and /sitemap.xml on 30.7% of 680, which is one origin more. A sitemap is addressed to search crawlers, not agents. Nearly nine in ten of the origins refusing one refuse the other.
So the effect is not about AI. /robots.txt is uniquely permitted — refused on only 15.1% of 680 origins from the datacenter arm — almost certainly because it is a well-known path that CDN and WAF layers special-case. Every other document gets the default treatment.
The two-tier edge mechanism
Tier 1 — the base edge rule. A client presenting a browser’s TLS handshake, claiming nothing about being an agent and asking for the identical files, already meets an elevated refusal rate away from /robots.txt: +6.5 points on the datacenter arm, +5.1 on the consumer one. The path is special-cased for every client type.
Tier 2 — the identity penalty. A declared AI user-agent pays that and a penalty on top, at +14.7 points and +13.7 — roughly double. Our first draft reported only the second number and read all of it as an AI effect. About half of it is not.
What each endpoint actually returned.
Declared agent, datacenter arm, 680 origins. A 404 is an answer, not a refusal — and for /llms.txt it is the ordinary case. Counting those as refusals is the single mistake that would have manufactured this study’s entire finding; section 3 shows the check. And a 200 is served, not adopted: a soft-404, a challenge wall or a sign-in page arriving with the success code is a page, not the file, and the of which rows below say how often that happened. Corrected 12 September 2026; section 3 records it.
| endpoint | distribution | served | absent | refused |
|---|---|---|---|---|
| /robots.txt | 559 · 82.2% | 15 · 2.2% | 89 · 13.1% | |
| /sitemap.xml | 269 · 39.6% | 199 · 29.3% | 190 · 27.9% | |
| /llms.txt | 130 · 19.1% | 339 · 49.9% | 189 · 27.8% |
| outcome | /robots.txt | /llms.txt | /sitemap.xml |
|---|---|---|---|
| 200 served | 559 · 82.2% | 130 · 19.1% | 269 · 39.6% |
| of which the document itself | 547 · 80.4% | 74 · 10.9% | 228 · 33.5% |
| of which a page, or empty | 12 · 1.8% | 56 · 8.2% | 41 · 6.0% |
| other served — 2xx/3xx, chiefly 202 | 3 · 0.4% | 3 · 0.4% | 3 · 0.4% |
| 404 absent — an answer | 15 · 2.2% | 339 · 49.9% | 199 · 29.3% |
| 403 refused | 75 · 11.0% | 150 · 22.1% | 153 · 22.5% |
| 402 refused | 2 · 0.3% | 17 · 2.5% | 11 · 1.6% |
| 429 refused | 3 · 0.4% | 7 · 1.0% | 11 · 1.6% |
| other refused — 406, 451, 500, 503 | 9 · 1.3% | 15 · 2.2% | 15 · 2.2% |
| no response — transport | 14 · 2.1% | 19 · 2.8% | 19 · 2.8% |
The rule is the path, not the address
If datacenter IP reputation drove this, the AWS arm would be the worse of the two. It is the better one on every endpoint, and the gap survives with the network held constant inside each arm.
| endpoint | datacenter · AS16509 | consumer · AS5483 | gap vs /robots.txt |
|---|---|---|---|
| /robots.txt | 13.1% | 15.6% | — |
| /llms.txt | 27.8% | 29.3% | +14.7 / +13.7 |
| /sitemap.xml | 27.9% | 29.0% | +14.9 / +13.4 |
What declaring an identity costs, path by path
The gap column above reads across a row — one client, two paths. Reading down instead gives the other effect: same path, same handshake, and the only difference is that one client says what it is.
| path | client | refusal | refused | identity penalty |
|---|---|---|---|---|
| /robots.txt | browser TLS handshake | 24 · 3.5% | ||
| declared AI agent | 89 · 13.1% | +9.6 pts | ||
| /llms.txt | browser TLS handshake | 68 · 10.0% | ||
| declared AI agent | 189 · 27.8% | +17.8 pts |
And the same origins do it on both networks. 81 origins from the datacenter arm and 73 from the consumer arm serve /robots.txt and refuse both other endpoints, and 71 of them are the same origins — a Jaccard index of 0.86 over the two sets. Address reputation would not select the same sites from two unrelated networks. On most of them the same server header issues the 200 and the refusal, so it is one layer applying a path rule rather than two systems disagreeing.
Three origins serve a declared agent a different policy than they serve a browser.
This result is not from the AIR study. It comes from the measurement of 26 August, which this study supersedes and which is not itself published here, and it is reproduced in full because it is that study’s sharpest finding and because the disclosure notes name this page as where the affected publishers are identified. Retiring a study must not retire a result.
The funnel is the finding’s own control. Sixteen per cent of comparable origins differ at the byte level — timestamps, cache keys, generated comments — and almost all of it evaporates by the level that maps to behaviour. Reporting the byte-level number as “the policy differs by client” would have inflated this result twentyfold.
Three origins are established as a delivery-layer effect: www.forbes.com, www.rollingstone.com and www.blic.rs. Each served the declared agent a 597-byte file that is byte-identical across five unrelated publishers in four countries, names none of them, and carries no Sitemap: directive — while every other client received the site’s own, entirely different file, within the same seconds. A static file cannot do that.
What layer answers the agent’s request is not established, and this page names no such layer. Nothing here attributes the behaviour, or any intent, to the publishers. The simpler reading is that the agent’s request is answered by a different service than the one answering browsers, and that service served its own default.
The full chain of evidence, the header analysis behind it and the two origins removed from an earlier count of five are in reports/cp-pub/attribution.md.
One origin in nine has written the file. Education refuses most; government has barely started.
The corpus is stratified into eight classes, and the path rule does not fall evenly across them. Adoption of /llms.txt — a 200 whose body is the file — is 74 of 680 on the datacenter arm, and it spans a factor of eighteen between SaaS and government. The sector most people expect to be strictest — news, with its licensing deals and its WAFs — sits in the middle of the refusal range rather than at the top of it.
| stratum | origins | outcome | file | 200 | refused |
|---|---|---|---|---|---|
| SaaS & B2B | n=63 | 21 · 33.3% | 23 · 36.5% | 12 · 19.0% | |
| Reference | n=79 | 11 · 13.9% | 14 · 17.7% | 13 · 16.5% | |
| E-commerce | n=108 | 15 · 13.9% | 35 · 32.4% | 36 · 33.3% | |
| Corporate | n=31 | 3 · 9.7% | 7 · 22.6% | 3 · 9.7% | |
| News | n=180 | 13 · 7.2% | 24 · 13.3% | 56 · 31.1% | |
| Community | n=101 | 7 · 6.9% | 18 · 17.8% | 32 · 31.7% | |
| Education | n=61 | 3 · 4.9% | 4 · 6.6% | 22 · 36.1% | |
| Government | n=57 | 1 · 1.8% | 5 · 8.8% | 15 · 26.3% |
This chart read differently until 12 September 2026. It counted a 200 as adoption, and 56 of 680 were not the file: a storefront answering every path with the shop, a “page not found” sent with the wrong status, a challenge wall. E-commerce looked like the second-largest adopter; most of that was its catch-all. The 200 column is kept so the old reading can be checked against the new one, and section 3 records the correction.
Publishing and refusing are still not opposites here. E-commerce is joint second in adoption and second in refusal at the same time: the stratum that writes the file is also the one most likely to have an edge rule that will not hand it over. What separates the strata is not a policy about AI, it is how much machinery sits in front of the origin.
Run again on 12 September, four profiles moved by less than a point. The declared agent moved by four — and not where the next change is coming from.
The same 680 origins, the same five profiles, the same pages — pinned to the 29 August draw rather than rediscovered — from the same datacenter network, fourteen days later. This run is the baseline for the one after it: on 15 September 2026 one edge vendor’s defaults for agent traffic change, per Cloudflare’s own announcement of 1 July 2026 on blog.cloudflare.com, and 205 of these origins carried that vendor’s response headers in August. Those 205 of 680 are the control stratum below; the other 475 of 680 are the rest. The design, the stratum membership and the decision rule were fixed in writing before this run started.
| endpoint | 29 August | 12 September | delta | flips from one edge product |
|---|---|---|---|---|
| /robots.txt | 15.1% | 11.9% | −3.2 | 13 of 19 |
| /llms.txt | 30.6% | 26.9% | −3.7 | 12 of 20 |
| /sitemap.xml | 30.7% | 26.8% | −4.0 | 13 of 22 |
The instrument held. Twelve of the fifteen endpoint-by-profile cells moved by at most 0.6 points — the floor a five-run repeat measured on this corpus a fortnight earlier — and both mechanisms in section 1 came back where they were: the base edge rule at +6.3 points, the identity penalty at +14.6. The remaining three cells are all the declared agent, all in the same direction, and mostly the same origins: roughly two in three of the sites that refused it in August and served it in September had returned the same edge product’s headers in August, and it is not the product whose defaults move on the 15th. The undeclared client on the same transport did not move, so the change is in how a class of edges treats the declaration, not in our client. Which setting moved, and at whose hand, is not something this page can see, and it does not guess.
In the control stratum, nothing moved — and it was already the strict one. Pages refused to the declared agent on the datacenter arm: 42.4% of 719 in the control stratum in August, 42.4% of 719 in September; in the rest, 19.4% of 1,492 then 16.9% of 1,505. A client that declares nothing is refused 32.7% of 724 pages in the control stratum and 8.5% of 1,488 in the rest, on the datacenter arm. Whatever the 15th does, that four-to-one gap is where it starts from. The run after the change is scheduled for the last week of September and will be reported here against these numbers, under the rule already written down.
Three origins in 680 publish an MCP manifest. One publishes an OpenAPI description at its root. Seven publish the long form of llms.txt.
Single-URL scanners score a site on whether it serves a handful of documents “for agents”: an MCP manifest, the retired ChatGPT plugin manifest, an OpenAPI description at four conventional paths, Markdown on request. Nobody had a stratified prevalence for any of them. The 12 September run asked all 680 origins, under three profiles from two networks, and read every 200 to see whether it was the document or a page wearing the status.
| path | document | page | absent | refused |
|---|---|---|---|---|
| /.well-known/mcp.json | 3 · 0.4% | 47 · 6.9% | 440 · 64.7% | 173 · 25.4% |
| /.well-known/ai-plugin.json | 4 · 0.6% | 51 · 7.5% | 436 · 64.1% | 171 · 25.1% |
| /llms-full.txt | 7 · 1.0% | 55 · 8.1% | 420 · 61.8% | 184 · 27.1% |
| /openapi.json | 1 · 0.1% | 55 · 8.1% | 424 · 62.4% | 185 · 27.2% |
| /openapi.yaml | 2 · 0.3% | 58 · 8.5% | 402 · 59.1% | 204 · 30.0% |
| /swagger.json | 1 · 0.1% | 53 · 7.8% | 419 · 61.6% | 190 · 27.9% |
| /api-docs | 0 · 0.0% | 57 · 8.4% | 413 · 60.7% | 194 · 28.5% |
The surface is nearly empty, and the edge treats it like everything else. Every one of these paths is refused to the declared agent on a quarter to a third of origins — the same default treatment section 1 found for /llms.txt and /sitemap.xml — and on fifty-odd origins per path a scanner reading the status would record a document that is a “page not found”. Asked for Markdown by Accept header, the homepage came back as Markdown on 15 of 469 origins that answered the declared agent on the datacenter arm; by query parameter, 1 of 469. The consumer arm reads the same documents, with refusal running two to three points higher — 28.5% of 680 for the MCP manifest on the residential arm AS6730.
Add it all up and the agent-native web is seventeen origins. Any of the seven documents at all: 17 of 680 on the datacenter arm. Count an llms.txt file as well and it is 90 of 680 — one origin in eight has written anything for an agent. The three MCP manifests belong to a Vietnamese news site, a British retailer and a health publisher; every one of the seven long-form llms.txt files sits beside a short one.
The soft-404 is one behaviour, not seven. 33 of 680 origins answer every one of the seven paths with a page wearing a 200 — storefronts and app shells that route any path to the front door — and 31 of the 54 page-200s at /llms.txt in section 2 are those same origins. A scanner reading statuses would credit each of them with eight documents.
The file and the edge disagree in both directions. On the datacenter arm, 444 of 569 served robots.txt files permit the AI-retrieval class at the root, and the edge refused the declared agent /llms.txt anyway on 48 of those 444 — the policy contradicted by delivery, with 27 of the 48 in the control stratum. Of the 125 of 569 that disallow it, the edge served the declared agent regardless on 75 of those 125. A disallow is a request, not a control, and most sites never wire the one to the other. The file’s rule is read, never sent.
None of this enters a score. A scanner that puts a quarter of its grade on this category is grading the web on a convention that seventeen origins in 680 follow, and reading a challenge wall as a failing manifest on fifty more.
What the corpus declares to named crawlers
Read off the robots.txt each origin served the declared agent — 569 files on the datacenter arm that are files and not pages. No request was ever made under any of these names; the table is what the files say, by stratum, for the two tokens people write most.
| stratum | files | GPTBot disallowed at / | ClaudeBot disallowed at / |
|---|---|---|---|
| News | n=157 | 65 · 41.4% | 73 · 46.5% |
| Community | n=79 | 32 · 40.5% | 27 · 34.2% |
| Education | n=48 | 13 · 27.1% | 5 · 10.4% |
| E-commerce | n=89 | 12 · 13.5% | 9 · 10.1% |
| Corporate | n=29 | 3 · 10.3% | 3 · 10.3% |
| Reference | n=67 | 5 · 7.5% | 4 · 6.0% |
| Government | n=42 | 3 · 7.1% | 3 · 7.1% |
| SaaS & B2B | n=58 | 2 · 3.4% | 1 · 1.7% |
Names, not policies. News blocks both and blocks the second harder; education blocks the first at nearly three times the rate of the second. Across the 569 files on the datacenter arm, 34 of 569 shut out GPTBot while admitting ClaudeBot, and 24 of 569 do the reverse. And 42 of 569 name an AI crawler for no other purpose than to allow it — the file’s vocabulary used as an invitation, on more sites than publish a manifest.
Corrected before publication, 12 September 2026: the /llms-full.txt row first read 61 documents. The reader accepted any text of a hundred characters as a thin file, and an HTML page is text; 55 of the 61 were pages. The same defect as section 3’s, caught by re-reading the stored bodies rather than trusting the row.
Three defects: two that would each have produced this finding out of nothing, and one that overstated a number beside it.
404 disambiguation — an absent file is an answer
/llms.txt genuinely does not exist at 339 of 680 origins on the datacenter arm. It is a new convention and most sites have not adopted it. Had those 404s been classified as refusals, the endpoint’s refusal rate would read close to three times what we publish, and the whole finding would be an artefact of an optional file being absent rather than evidence of anything.
Zero of those 339 are counted as refusals — verified across all three endpoints on both networks, and asserted in the generated report rather than trusted to a code comment. It is the single decision this study most depends on, so it fails loudly if it ever regresses.
The parser read challenge walls as permission
A separate measure asks how often the edge refuses a URL that the site’s own robots.txt permits. It was first published at roughly twice the rate reported below, and the difference was entirely this defect.
84 origins answered the agent’s request for /robots.txt with an HTTP 403 challenge page, and the harness stored that HTML as though it were the policy. A robots parser reads a page of markup as a ruleset containing no rules — which means everything is permitted. The origin’s own refusal then became evidence that it contradicts its own published policy.
Corrected to count only origins that actually served the file, the rate is 8.9% of 1,902 agent page requests on the datacenter arm and 9.4% of 1,732 on the consumer arm. An origin that refuses you its policy has published no rule you could have obeyed, and counting it as a contradiction was the same error in a different coat.
A 200 is served, and it is not the file — corrected 12 September 2026
The outcome table and the by-stratum chart counted a 200 at /llms.txt as adoption. Reading the stored bodies: 56 of 680 origins on the datacenter arm answered with a 200 whose body is an HTML page or nothing — “Page not found”, a challenge wall, a sign-in form, a single-page app’s shell, a storefront that answers every path with the shop. The file exists at 74 of 680, not 130. The same is true at /sitemap.xml, where 41 of 680 200s are not a sitemap.
The status rows are unchanged and still sum to 680; two of which rows now split them by what the body was. The by-stratum chart orders and colours by the file, keeps the status count beside it, and its headline changed: adoption spans a factor of eighteen rather than five, and government, not education, publishes least. No refusal figure moved — a page served with a 200 was never counted as a refusal, and is not now.
It is the defect above, one status code over: a wall stored as a 403 was read as a permissive ruleset; a page stored as a 200 was read as an adopted file. The bodies were kept, which is why this could be found and fixed without asking 680 origins again.