/robots.txt is the exception. Everything else is refused at twice the rate.
A differential study of what an origin serves a declared AI agent against what it serves a browser, across two networks and five client identities. It separates how much of an agent’s disadvantage comes from being an agent from how much comes from asking for a path the web was not built to serve — and finds that about half of it is not about AI at all.
Read the findings · The long-form report · Every figure as JSON