The artificial barrier, quantified — and about half of it is not artificial.
Origin policies, dual-egress routing and client identity, measured across 680 domains from two networks under five client profiles. Web servers do treat declared agents differently. They also treat every non-standard path differently, and telling those two apart is most of the work.
The thesis, in one paragraph. Edge networks special-case /robots.txt. It is a well-known path and it is allowed through. Ask the same infrastructure for /llms.txt or /sitemap.xml and the default rule applies instead — to a browser as well as to an agent. Declaring yourself an AI agent then adds a second, separate penalty on top. The two are additive, and only the second is about AI.
Two-factor differential protocol
Two factors vary and everything else is held still: which network the request leaves from, and what the client claims to be. The five profiles are interleaved per origin inside a one-second-per-host pacing window, so all five readings of one file happen within seconds of each other and a difference between them cannot be elapsed time.
/llms.txt responses are 404, and folding those into refusals would multiply that endpoint’s apparent hostility by about four.The five profiles
No third-party crawler’s name was ever sent. Presenting as GPTBot or any other operator’s agent would put traffic in their logs that they did not send, so the profiles either present a browser honestly or declare what they actually are. The two variables under test — the claimed identity and the TLS handshake — are isolated without it.
Browser, scripted
browser_js · Chromium, full render
The reference. What a site serves a real browser, with JavaScript executed.
Browser, no JavaScript
browser_nojs · browser UA, plain client
Claims to be Chrome, handshakes like Python. Isolates the contradiction between UA and TLS.
Browser, impersonating handshake
browser_nojs_impersonate · real Chrome TLS
A genuine browser fingerprint without JavaScript. The baseline for the path penalty.
Declared agent
agent_fetch · names itself, publishes a contact URL
Honest about being an automated agent. Measures what declaring costs.
Minimal client
plain_fetch · no browser pretence
A generic HTTP client. Separates identity discrimination from bot heuristics.
Why two networks. A refusal can be about the address or about the client. Running the identical matrix from a datacenter ASN and a consumer mobile carrier separates them — and it did: the gap between /robots.txt and the other paths is the same size on both, and the datacenter arm is the less refused of the two, which is the opposite of address-reputation scoring.
Endpoint response distribution
Declared agent, datacenter arm, of 680 origins.
| endpoint | 200 served | other 2xx/3xx | 404 absent | refused | no response |
|---|---|---|---|---|---|
| /robots.txt | 559 · 82.2% | 3 · 0.4% | 15 · 2.2% | 89 · 13.1% | 14 · 2.1% |
| /sitemap.xml | 269 · 39.6% | 3 · 0.4% | 199 · 29.3% | 190 · 27.9% | 19 · 2.8% |
| /llms.txt | 130 · 19.1% | 3 · 0.4% | 339 · 49.9% | 189 · 27.8% | 19 · 2.8% |
Which refusal
“Refused” above is every status that turns a client away except 404. Broken out, one mechanism dominates and the rest are marginal.
| endpoint | 403 forbidden | 402 payment required | 429 rate limited |
|---|---|---|---|
| /robots.txt | 75 · 11.0% | 2 · 0.3% | 3 · 0.4% |
| /sitemap.xml | 153 · 22.5% | 11 · 1.6% | 11 · 1.6% |
| /llms.txt | 150 · 22.1% | 17 · 2.5% | 7 · 1.0% |
403 is the firewall’s answer and everything else is a rounding error beside it. The three columns do not sum to the refused total, and that is deliberate: other statuses turn a client away too, and they are counted in the aggregate rather than dropped from it. A study that reports only these three undercounts refusal.
The 404 stratum is the largest single outcome for /llms.txt, at 339 of 680 origins. That is not blocking. It is a convention most of the web has not adopted, and reporting it as hostility would be the most consequential error available here.
By stratum
The corpus’s own eight strata. Served counts a 200 only — an absent file is a valid answer but it is not adoption, and the two must not be summed.
| stratum | origins | /llms.txt served | /llms.txt refused | /robots.txt refused |
|---|---|---|---|---|
| news | 180 | 13.3% of 180 | 31.1% of 180 | 9.4% of 180 |
| commerce | 108 | 32.4% of 108 | 33.3% of 108 | 19.4% of 108 |
| community | 101 | 17.8% of 101 | 31.7% of 101 | 13.9% of 101 |
| reference | 79 | 17.7% of 79 | 16.5% of 79 | 10.1% of 79 |
| saas | 63 | 36.5% of 63 | 19% of 63 | 9.5% of 63 |
| education | 61 | 6.6% of 61 | 36.1% of 61 | 16.4% of 61 |
| government | 57 | 8.8% of 57 | 26.3% of 57 | 19.3% of 57 |
| corporate | 31 | 22.6% of 31 | 9.7% of 31 | 6.5% of 31 |
Education refuses the agent /llms.txt most, and publishes it least. SaaS is the inverse on both counts. The spread is real and it does not line up with the intuition that news publishers are the strictest — on this endpoint they are mid-pack.
Two penalties, and they add
The path is special-cased
A client presenting a genuine browser handshake, claiming nothing about being an agent, still meets +6.5 points more refusal away from /robots.txt. Well-known paths are allowed through; everything else meets the default rule.
The identity costs extra
The declared agent pays that penalty and another on top: +14.7 points, roughly double. Only this second term is about being an AI agent, and a study reporting the total as an AI effect overstates it about twofold.
The same origins do it on both networks. 81 origins from the datacenter arm and 73 from the consumer arm serve /robots.txt and refuse both other endpoints, and 71 of them are the same origins. On most, the same server header issues the 200 and the refusal — one layer applying a path rule, not two systems disagreeing.
The schema and the prose disagree
A status code does not settle whether content arrived. On URLs where the agent and the browser were both served a 200, the agent sometimes received a fraction of the text — and when that happened, the structured data almost always survived intact.
The two layers then contradict each other. The schema still declares an Article; the prose no longer contains one. An agent that trusts structured data — which is what structured data is for — would conclude it received something it did not.
What this run cannot tell you about 402
402 Payment Required appears on 17 of 680 origins for /llms.txt. It is tempting to report that none of them offered a machine-actionable payment handshake, and an earlier draft of this report did.
That claim was our own instrument, not the corpus. The header most likely to carry a payment challenge was on a never-store list when these rows were written, so it could not reach the classifier that looks for one. Its absence is our doing. The question is open, and it is recorded that way rather than answered.
What would change our mind
This report is one run. Its figures are re-derivable from stored response rows, and every one of them is registered with its numerator, denominator, egress arm and the section it was read off — as machine-readable JSON.
The load-bearing decision is treating a 404 as an answer. If that is wrong, the /llms.txt result is wrong by a factor of about four, and it is the first thing a sceptical reader should attack. The second is the split between the path penalty and the identity penalty: it rests on a browser profile meeting the same elevated refusal on the same paths, and if that profile is not a fair browser baseline, the split moves.
The summary of this study, with the differing-policy finding it carries forward, is the findings page. The method, corpus and client definitions are on the method page.