/robots.txt is the exception. Everything else is refused at twice the rate.
A differential study of what an origin serves a declared AI agent against what it serves a browser, across two networks and five client identities. It separates how much of an agent’s disadvantage comes from being an agent from how much comes from asking for a path the web was not built to serve — and finds that about half of it is not about AI at all. Repeated on 12 September on the same pages: four profiles moved by under a point, the declared agent by four, and the agent-native surface — MCP manifests, OpenAPI descriptions, Markdown on request — turned out to be nearly empty.
Read the findings · The long-form report · Every figure as JSON