AgentProbe

AgentProbe is a web crawler operated by Aron Horvath, an independent researcher. It measures how readable and operable public web pages are to AI systems — search crawlers, retrieval pipelines and agentic browsers — and publishes what it finds. So far that is one study of 680 origins. If it reached your site and you would rather it did not, the next section is the whole answer.

Blocking it

Add this to your robots.txt:

User-agent: AgentProbe
Disallow: /

We read robots.txt fresh at the start of every crawl and never cache a previous answer, so a block takes effect the next time we look at your site. There is nothing to sign up for and no confirmation step.

To allow some paths and not others, ordinary robots.txt rules apply:

User-agent: AgentProbe
Disallow: /members/
Disallow: /search
Allow: /

Identifying it

Every request carries this exact user-agent string:

AgentProbe/0.1.0 (+https://agent-probe.com/bot)

Token

AgentProbe — the only name we crawl under

Source addresses

Not published, and there is no range to publish. The one study run so far went out from a single cloud instance in AWS eu-west-1 (AS16509) and, for a smaller paired arm, a consumer ISP line in Switzerland (AS6730). The cloud address changes every time that host is stopped and started, and the residential one is a person's home connection. Neither is a stable range anyone could allowlist, and publishing the second would identify an individual. Verify us by the user-agent string, not by address — and see the paragraph below on why that is weaker than it sounds.

Signed requests

Not yet. When we adopt Web Bot Auth, the public key will be published here and under /.well-known/, and this line will say so.

Which means the two lines above are an admission: right now you cannot cryptographically verify that a request calling itself AgentProbe is ours. Anyone can send any user-agent string. If something claiming to be us is doing anything this page says we do not do — ignoring robots.txt, hammering you, fetching images, touching forms — then it is not us, and we would like to know: bot@agent-probe.com. Blocking the token costs you nothing either way, because a block is a result we publish rather than an obstacle we work around.

We never send requests under another crawler's name. When our reports say a site permits or blocks GPTBot, ClaudeBot, PerplexityBot or any other crawler, that is derived by parsing your published robots.txt against those tokens — not by impersonating them to see what happens. If you see traffic claiming to be a different crawler, it is not us.

What it fetches

One audit of one origin is small and deliberately slow. It samples a handful of pages rather than crawling a site exhaustively, because the goal is to characterise a site, not to copy it.

Per audit

About 10 requests to your origin, covering up to 5 pages

Concurrency

One request at a time per origin — never parallel against you

Spacing

At least 1 second between requests to the same origin

Duration

Around 15 seconds from first request to last

Also fetched

/robots.txt, your sitemap, /llms.txt and /llms-full.txt if they exist

Never fetched

Images, video, audio and fonts — aborted at the network layer

That last one is most of the bytes. We measure text and markup, so during rendering we abort every image, media and font request before it leaves the browser. You can verify it in your own access logs: you should see our HTML requests and none of the assets those pages reference.

Pages are fetched twice: once as raw HTML, and once in a real headless Chromium browser, which executes your JavaScript. The difference between the two is the main thing we measure — most AI crawlers do not run JavaScript, so content that only appears after it runs is invisible to them. We say so plainly because a rendered fetch shows up differently in your analytics than a plain one.

What it does not do

• It does not submit forms, log in, or follow links behind authentication.
• It does not attempt purchases, bookings, or any other state-changing action.
• It does not try to evade bot protection. If you challenge us, we record that we were challenged and move on — a refusal is a finding we publish, not an obstacle we route around.
• It does not crawl continuously. This is a research crawler, and it has run one study: 680 origins, measured over a single window in August 2026. There is no on-request audit and no recurring benchmark — if it reached your site, it was as one origin in that study's population.
• It does not sell, republish or train models on your content.

What it keeps

We store byte-exact copies of the pages we fetch, addressed by content hash, so that a score can be recomputed later without crawling you again. That is the point: when our scoring model changes, we re-read what we already have rather than coming back.

Stored copies are used to derive measurements, and to show an operator what we saw on their own origin if they ask. They are not published, not redistributed, and not used as training data. Nothing is served from them: what gets published is the derived figures and the method that produced them. To have captures of your origin deleted, email bot@agent-probe.com.

Why it exists

Sites are increasingly read by machines rather than people, and most site owners have no way to see what those machines actually get. We measure it, publish the method and the weights, and let anyone check the result against their own site. The full method, including every signal and its weighting, is published at agent-probe.com/method, and the one study run so far is at agent-probe.com/findings/agent-access-2026-08.

Contact

Anything about this crawler

bot@agent-probe.com

Operator

Aron Horvath, an independent researcher. Not a company; there is no support desk behind this address, only a person who reads it.

If this crawler caused a problem on your site — load, cost, log noise, anything — tell us and we will fix it. That is the entire reason this page has an address on it.