Originally published on mrsaynothing.dev — every post there passes a content gate and prints its provenance.
Seven nights of door knocks on this site: 8,654 crawler hits against 247 human visits — 35 knocks per reader. The AI-crawlers-vs-search-crawlers split is already real: PerplexityBot out-crawled OpenAI's OAI-SearchBot 273 to 30. SEO-tool crawlers (1,274 knocks) out-crawled the search engines' own bots (885). And 65% of all knocks arrived with no name. The full census is below — so is the ten-minute way to run it on your own logs.
A week of crawler traffic on a live, small site (7 nights ending 2026-11-01): 8,654 bot hits, only 2,986 of them named, across 24 distinct bots. The loudest search-adjacent family is already AI: PerplexityBot alone made 273 knocks vs OAI-SearchBot's 30. The search engines' crawlers together (Yandex 643, DuckDuckBot 137, Bingbot 91, Googlebot 14) total 885 — beaten by the SEO-tool crawlers' 1,274. 65.5% of hits identify no bot at all.
A crawler census is the guest list nobody publishes.
What are AI crawlers?
Claim: an AI crawler is a bot that reads your pages so a model can quote, cite or learn from them — and it announces itself the same way every bot has since the nineties: a user-agent string. Four words carry this whole census:
- Crawler — a program that fetches pages, follows links, and comes back. Picture a courier who photographs every page of your notebook and files the copies.
-
User-agent — the name the crawler prints when it knocks.
PerplexityBot/1.0is a name tag; an empty or generic string is a drawn hood. - robots.txt — the sign on the door listing which names may enter. Every big AI crawler honours it; that is exactly how you can count them.
- Census — the count of who actually knocked, not who was allowed to. Rules on paper vs the lobby log.
The distinction that matters in 2026 is purpose, not name. Search crawlers index so a human can find you later. AI-search crawlers fetch so a model can answer with you today. AI-training crawlers fetch so a future model can speak in your voice. Three different reasons to knock, three different values to you.
AI crawlers vs search crawlers
Claim: on this site, for this week, the AI family already out-knocks every search engine except Yandex — and the loudest crawlers of all are neither: they are SEO tools. The full census:
| Bot | Operator | Purpose | Knocks (7d) |
|---|---|---|---|
| MJ12bot | Majestic | seo | 655 |
| YandexBot | Yandex | search | 643 |
| SemrushBot | Semrush | seo | 442 |
| PerplexityBot | Perplexity | ai_search | 273 |
| AhrefsBot | Ahrefs | seo | 177 |
| Amazonbot | Amazon | ai_training | 167 |
| ClaudeBot | Anthropic | ai_training | 151 |
| DuckDuckBot | DuckDuckGo | search | 137 |
| DuckAssistBot | DuckDuckGo | ai_search | 122 |
| Bingbot | Microsoft | search | 91 |
| ChatGPT-User | OpenAI | ai_search, human-triggered | 32 |
| OAI-SearchBot | OpenAI | ai_search | 30 |
| Googlebot | search (named from Nov 1) | 14 | |
| Bytespider | ByteDance | ai_training | 13 |
| — no name given | — | — | 5,668 |
Read it by family and the week says three things. SEO-tool crawlers — Majestic, Semrush, Ahrefs — knocked 1,274 times, more than every search engine combined (885). These are bots that read your site to sell you ranking reports. The AI family — ai_search plus ai_training — knocked 788 times: Perplexity 273, Amazon 167, Anthropic 151, DuckDuckGo's assistant 122, OpenAI's pair 62, ByteDance 13. PerplexityBot alone out-crawled OpenAI's OAI-SearchBot 273 to 30 — nine to one.
And the human counterpoint that gives the census its scale: 247 human visits in the same seven nights. Thirty-five bot knocks for every actual reader. The site is mostly read by machines that will never scroll, never bounce, never buy — and the guest list nobody prints is two-thirds anonymous.
Why do two-thirds of knocks have no name?
Claim: 65.5% of this week's knocks — 5,668 of 8,654 — carried no bot name at all, and that number is a floor, not a ceiling, because name detection only catches what it knows. We call the share of bot traffic that identifies itself the name-tag rate; this week it was 34.5%.
Three kinds of guest wear the hood. Headless browsers driving scrapers print the browser's name, not the company's. Fresh bots launch before detection lists learn them. And some names are simply rare enough that generic rules — "anything matching bot, spider, crawl" — count them without naming them. The census can only name what its list knows.
What changed in our own count this week
Until this week, several big crawlers — Googlebot, Bingbot, Applebot — matched only generic rules in the detection list this site uses, so they landed in the unnamed pile like everyone else. On 2026-11-01 we added them as named entries. That is why Googlebot's row above reads one day, not seven, and why we print the caveat instead of quietly comparing seven nights of Perplexity against one night of Google.
Should you block AI crawlers?
Claim: on the numbers, the AI crawlers are not the ones worth a robots.txt fight — the SEO tools are. The entire GPT-family footprint here for the week was 62 knocks. The three SEO-tool crawlers knocked 1,274 times, and the loudest guest of all, MJ12bot at 655, reads sites to feed a link-intelligence database. MJ12, incidentally, is the UFO-conspiracy office from The X-Files — someone named their link-indexing crawler after a secret government file, which might be the most honest branding in SEO.
We allow the AI family, deliberately. This site is built to be quoted: every answer card carries a stable anchor so a model can cite exactly the sentence it used, and the AI referral channel here grew 50% week over week — from a small base, but from the direction we want. Blocking Perplexity saves 273 fetches a week and costs us the fastest-growing reader we have. SemrushBot checking our rankings 442 times — for queries most humans never type — is a harder trade to defend.
When the knocks get heavy enough to matter, the door itself needs arithmetic. We have opinions about that too — it is the same counting problem behind the rate limiter that ate our test suite.
How to run your own crawler census
Claim: you need nothing but your access log and ten minutes. The same four steps run on any server log, CDN dashboard or analytics bot tab:
- [ ] Find the raw log or the bot tab — wherever requests land with their user-agent.
- [ ] Count the named AI bots with one grep.
- [ ] Count total requests, compute the name-tag rate.
- [ ] Repeat weekly — the trend is the story, not the snapshot.
# who knocked, by name
grep -oE '(GPTBot|OAI-SearchBot|PerplexityBot|ClaudeBot|Claude-Web|Googlebot|Bingbot|Applebot)' access.log \
| sort | uniq -c | sort -rn
# the name-tag rate: bot-shaped requests vs everything
grep -cE '(bot|spider|crawl)' access.log
wc -l < access.log
The greps were run against a 10-line sample log on 2026-11-01 — the first command prints one count per name, the second prints the bot-shaped total. Numbers in the census itself come from this site's own bot log, pulled the same day.
What the next census will say
Claim: on this site the interesting row is the one at the bottom of the search family. Googlebot's named count is 14, and Google referrals have been flat zero for eleven straight days on our stats — a migration renamed a few hundred URLs, the redirects pointed at doors that no longer existed, and Google quietly stopped knocking while it re-learned the floor plan. The fix went up this morning; whether the guest list recovers is exactly what next week's census will show.
We will keep the door open and the list public. A site that hides its census is just a site with a hood on.
Census window 2026-10-25 → 2026-11-01, pulled 2026-11-01, last tested 2026-11-01.
Your turn: which bot surprised you most in your own logs — and have you ever managed to name the anonymous two-thirds? The honest answers go in next week's census.
Top comments (18)
@313automationspak Good catch — I re-ran the census tonight to answer this properly: ChatGPT-User showed 46 hits in the same seven nights (Claude-User: 2). So the reader-triggered lane is real, just an order of magnitude under the crawler side — PerplexityBot sat at 379 by tonight. The post compares crawler to crawler; you are right that the user-triggered fetchers belong on the chart as their own curve, because they follow human questions, not a crawl budget. The next census revision splits the rows three ways: crawler, user-triggered, unnamed.
The row that surprised me: the SEO-tool trio (1,274 knocks) out-crawled every search engine combined (885). Majestic's MJ12bot alone out-fetched Googlebot's named count by roughly 47 to 1 — and MJ12 is named after the X-Files UFO office. Which of your crawlers would you actually miss if it stopped knocking?
@alexshev The audit-trail point stands. What we store per knock today is UA, path and timestamp — the shape the dashboard gives us without keeping header dumps of every visitor on a public path. Your robots-response column is the one I can add cheapest and it is the most diagnostic: it separates a bot that ignored a transient rule from one that obeyed, which is exactly the doubt you name in the last line. Next revision: per-knock status code plus ASN grouping, both offline, both in the same table.
On naming the anonymous 65.5%: stop looking at the user agent and look at who owns the IP. Group those 5,668 hits by ASN and organisation and most of them get a name fast, usually a handful of cloud and hosting networks plus a long tail of home ISPs. Reverse DNS plus the published crawler IP lists then confirms which of your 24 named bots are real and which are borrowing the name.
You can do the ASN step offline with a free IP-to-ASN dataset. If you'd rather call an API, IPGeolocation.io (IPGeolocation.io/ip-location-api.html) returns ASN and organisation on the free plan, 1,000 lookups a day (I work on it). The hosting and proxy flags are paid, but for a 7-night census the ASN alone does most of the work.
@furqan_ashraf Right diagnosis — the UA names the polite, the IP names the rest. Late follow-up (a rough week on our side): the next census cut groups the unnamed by ASN offline from the free IP-to-ASN dataset first — the hits stay ours. Your API works for spot checks, but a 7-day window wants the offline pass. The reverse-DNS confirmation also got backup below (the Googlebot check): that pass is how we prove a borrowed name — our Bingbot count jumped the day the crawler returned named, and the ASN check is what tells us it was never a spoof.
Makes sense, keeping the hits local is the right call for a full week. One tip for the offline pass: sort the IP-to-ASN ranges once and binary-search each IP against them instead of looping per line, and the 7 nights will run in seconds. Looking forward to the next census, happy to help if anything in the ASN grouping looks odd.
Since you've kept the user-agent strings, could you rerun all seven nights through the updated detection list? That would give Googlebot and the other newly named bots a full week's count too. It would also show how much of the anonymous 65% was a gap in your list rather than crawlers withholding their identity.
The finding I keep coming back to is that 65% of knocks had no bot name at all. It makes "block or allow AI crawlers in robots.txt" feel like a smaller lever than people assume, since the unnamed traffic won't read it anyway. Did you try grouping that unnamed share by ASN or IP range to see how much of it is the same crawlers in disguise?
The nine-to-one PerplexityBot vs OAI-SearchBot gap is interesting. One thing worth adding before reading it as interest: ChatGPT-User only shows up when a person asks ChatGPT about a page, so counting it next to OAI-SearchBot gives a fuller picture of how often ChatGPT actually reads the site. Did it appear at all in the seven nights?
The search crawler bargain was an exchange of bandwidth for referral traffic. You covered the server ingress and egress, and Googlebot sent human visits that converted into readers or subscribers. That trade had positive expected value for the publisher.
AI crawlers break that payoff structure. When an answering engine fetches a page to synthesize an answer directly on its own surface, referral probability drops toward zero while the hosting and compute costs remain on your ledger. The publisher underwrites the retrieval context for someone else's subscription revenue.
The 65 percent unnamed share is the predictable equilibrium. Once publishers realize the net payoff of named bot visits is negative and update robots.txt, crawlers face an immediate incentive to strip their user-agent headers and route through residential proxy pools. A voluntary identification standard cannot survive when the economics behind it turn purely extractive.
@deanlee This is the frame the whole thread has converged on — the referral column was the deal, and the answering engines fetch exactly the pages that used to earn it. A week of comments here has sharpened the identity side (ASN grouping, reverse-DNS); the column I still cannot print is referrals per bot. The honest-column fix is specced for the next census revision: knocks in one column, humans delivered in another, arithmetic labeled as arithmetic. Until reindex hands search traffic back, "bandwidth for nothing" is what the unnamed 65.5% charge us.
A short census becomes much more useful if each observation records request headers, IP or ASN, robots response, route, and timestamp. That separates crawler identity from traffic shape and makes later policy changes auditable. It also avoids drawing a broad conclusion from one bot that happened to ignore a transient rule.
The 8,654 bot knocks vs 247 human visits ratio is the kind of number I wish more robots.txt debates started with. The Perplexity-vs-OpenAI gap and the unnamed 65% are especially useful — did the unnamed share look like scrapers with rotating UAs, or mostly empty/generic agents?