@313automationspak Good catch — I re-ran the census tonight to answer this properly: ChatGPT-User showed 46 hits in the same seven nights (Claude-User: 2). So the reader-triggered lane is real, just an order of magnitude under the crawler side — PerplexityBot sat at 379 by tonight. The post compares crawler to crawler; you are right that the user-triggered fetchers belong on the chart as their own curve, because they follow human questions, not a crawl budget. The next census revision splits the rows three ways: crawler, user-triggered, unnamed.
The row that surprised me: the SEO-tool trio (1,274 knocks) out-crawled every search engine combined (885). Majestic's MJ12bot alone out-fetched Googlebot's named count by roughly 47 to 1 — and MJ12 is named after the X-Files UFO office. Which of your crawlers would you actually miss if it stopped knocking?
@alexshev The audit-trail point stands. What we store per knock today is UA, path and timestamp — the shape the dashboard gives us without keeping header dumps of every visitor on a public path. Your robots-response column is the one I can add cheapest and it is the most diagnostic: it separates a bot that ignored a transient rule from one that obeyed, which is exactly the doubt you name in the last line. Next revision: per-knock status code plus ASN grouping, both offline, both in the same table.
I write practical, hands-on guides about APIs, DNS, and web security. I like taking topics that feel complicated and breaking them into clear steps you can test yourself.
On naming the anonymous 65.5%: stop looking at the user agent and look at who owns the IP. Group those 5,668 hits by ASN and organisation and most of them get a name fast, usually a handful of cloud and hosting networks plus a long tail of home ISPs. Reverse DNS plus the published crawler IP lists then confirms which of your 24 named bots are real and which are borrowing the name.
You can do the ASN step offline with a free IP-to-ASN dataset. If you'd rather call an API, IPGeolocation.io (IPGeolocation.io/ip-location-api.html) returns ASN and organisation on the free plan, 1,000 lookups a day (I work on it). The hosting and proxy flags are paid, but for a 7-night census the ASN alone does most of the work.
@furqan_ashraf Right diagnosis — the UA names the polite, the IP names the rest. Late follow-up (a rough week on our side): the next census cut groups the unnamed by ASN offline from the free IP-to-ASN dataset first — the hits stay ours. Your API works for spot checks, but a 7-day window wants the offline pass. The reverse-DNS confirmation also got backup below (the Googlebot check): that pass is how we prove a borrowed name — our Bingbot count jumped the day the crawler returned named, and the ASN check is what tells us it was never a spoof.
I write practical, hands-on guides about APIs, DNS, and web security. I like taking topics that feel complicated and breaking them into clear steps you can test yourself.
Makes sense, keeping the hits local is the right call for a full week. One tip for the offline pass: sort the IP-to-ASN ranges once and binary-search each IP against them instead of looping per line, and the 7 nights will run in seconds. Looking forward to the next census, happy to help if anything in the ASN grouping looks odd.
Since you've kept the user-agent strings, could you rerun all seven nights through the updated detection list? That would give Googlebot and the other newly named bots a full week's count too. It would also show how much of the anonymous 65% was a gap in your list rather than crawlers withholding their identity.
SEO monitoring tool. Crawls your site daily or hourly and alerts you when a noindex tag, robots.txt block, sitemap error or canonical change appears. Free plan.
The finding I keep coming back to is that 65% of knocks had no bot name at all. It makes "block or allow AI crawlers in robots.txt" feel like a smaller lever than people assume, since the unnamed traffic won't read it anyway. Did you try grouping that unnamed share by ASN or IP range to see how much of it is the same crawlers in disguise?
AI automation and software development agency in Lahore, Pakistan (since 2018). We build AI agents, chatbots, workflow automation, web and mobile apps, and do technical SEO and GEO.
The nine-to-one PerplexityBot vs OAI-SearchBot gap is interesting. One thing worth adding before reading it as interest: ChatGPT-User only shows up when a person asks ChatGPT about a page, so counting it next to OAI-SearchBot gives a fuller picture of how often ChatGPT actually reads the site. Did it appear at all in the seven nights?
The search crawler bargain was an exchange of bandwidth for referral traffic. You covered the server ingress and egress, and Googlebot sent human visits that converted into readers or subscribers. That trade had positive expected value for the publisher.
AI crawlers break that payoff structure. When an answering engine fetches a page to synthesize an answer directly on its own surface, referral probability drops toward zero while the hosting and compute costs remain on your ledger. The publisher underwrites the retrieval context for someone else's subscription revenue.
The 65 percent unnamed share is the predictable equilibrium. Once publishers realize the net payoff of named bot visits is negative and update robots.txt, crawlers face an immediate incentive to strip their user-agent headers and route through residential proxy pools. A voluntary identification standard cannot survive when the economics behind it turn purely extractive.
@deanlee This is the frame the whole thread has converged on — the referral column was the deal, and the answering engines fetch exactly the pages that used to earn it. A week of comments here has sharpened the identity side (ASN grouping, reverse-DNS); the column I still cannot print is referrals per bot. The honest-column fix is specced for the next census revision: knocks in one column, humans delivered in another, arithmetic labeled as arithmetic. Until reindex hands search traffic back, "bandwidth for nothing" is what the unnamed 65.5% charge us.
A short census becomes much more useful if each observation records request headers, IP or ASN, robots response, route, and timestamp. That separates crawler identity from traffic shape and makes later policy changes auditable. It also avoids drawing a broad conclusion from one bot that happened to ignore a transient rule.
The 8,654 bot knocks vs 247 human visits ratio is the kind of number I wish more robots.txt debates started with. The Perplexity-vs-OpenAI gap and the unnamed 65% are especially useful — did the unnamed share look like scrapers with rotating UAs, or mostly empty/generic agents?
Googlebot is the crawler I'd miss most, and the purpose column hides why: Google's AI Overviews and AI Mode answer from what plain Googlebot fetches, so those 14 knocks are search and AI at once.
Google has no separate AI user agent for a census to count. Google-Extended is a robots.txt token only; Google's crawler docs say it makes no requests of its own, and blocking it covers Gemini training and grounding, not AI Overviews. So splitting by user agent will undercount Google's AI demand however complete the detection list gets.
For the 14 itself, Search Console holds Google's side of the ledger: Settings, Crawl stats, which counts requests per day and splits them by Googlebot type. Since Googlebot was only named from Nov 1, compare that day's figure with yours. If Google's is far higher, the gap is your list, and the rerun Stefan asked for will close it. If they roughly match, Google is crawling you lightly, and that is the number worth watching, more than PerplexityBot's 273.
Two pushes on this one, both from the other side of the glass.
First: a chunk of your 65.5% unnamed pile probably isn't crawlers at all. On a small site, anonymous traffic is dominated by vulnerability scanners hammering /wp-login.php, /.env and /xmlrpc.php, plus credential-stuffing POSTs — they fetch one or two URLs, never follow links, never read content. Folding them into the same column as SemrushBot makes bot pressure look scarier than it is. A crude cut — unnamed requests that got 404/403, or asked for a non-page path — would likely shrink that 5,668 by half. My own noise floor agrees from an unexpected direction: the single biggest bot mess on my site turned out to be my own analytics plugin generating admin-ajax URLs, 46 of which robots.txt now blocks. Anonymous volume needs composition analysis before it counts as a census row.
Second: crawl volume is less a property of the bot than a verdict on the site. In Search Console, one of my sites had 30 URLs moved from indexed to "crawled - currently not indexed" in a single batch on Oct 16, and then Google re-crawled nothing for 15 days. Same robots.txt, same server, zero knocks — because it had already decided the pages weren't worth fetching. Your own table has the tell: Googlebot at 14 against Yandex's 643 isn't Yandex being nine times hungrier, it's Google fetching on a quality judgment while Yandex fetches on a schedule. A census of door knocks measures who still believes you're worth fetching; the decision happens before the knock.
Which makes the SEO-tool family (1,274 knocks) the really interesting row — those bots read your site to sell you ranking reports, and they don't stop when the quality verdict goes against you. The bots that knock most persistently are the ones monetizing your anxiety, not the ones sending you readers.
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. kevin@outricher.com
from the scraper side the user agent is the first thing anyone fakes, so the named count reads like a floor and not a share. whether a client asked for robots.txt before anything else probably splits that unnamed bucket better than the name does
The 65% unnamed bucket is the interesting part. Have you tried splitting it by ASN? In logs I've looked at, a big chunk of "no name" traffic comes from a handful of cloud/hosting ASNs, which is a decent hint it's scrapers rather than people. Also worth verifying the named ones by reverse DNS or published IP ranges, since spoofed Googlebot/GPTBot user agents are common. Googlebot at 14 hits feels low enough that I'd want to know whether some real Googlebot traffic landed in the unnamed bucket.
SEO monitoring tool. Crawls your site daily or hourly and alerts you when a noindex tag, robots.txt block, sitemap error or canonical change appears. Free plan.
Reverse-DNS the IPs claiming to be Googlebot before you trust the name tag. A chunk of the "named" pile is spoofed the same way the unnamed two-thirds is — boring check, still catches a surprising number of fakes in access logs. Your Googlebot=14 footnote is the other useful bit: a migration that left redirects pointing at gone URLs will quietly starve the one crawler that actually sends traffic.
We're a place where coders share, stay up-to-date and grow their careers.
@313automationspak Good catch — I re-ran the census tonight to answer this properly: ChatGPT-User showed 46 hits in the same seven nights (Claude-User: 2). So the reader-triggered lane is real, just an order of magnitude under the crawler side — PerplexityBot sat at 379 by tonight. The post compares crawler to crawler; you are right that the user-triggered fetchers belong on the chart as their own curve, because they follow human questions, not a crawl budget. The next census revision splits the rows three ways: crawler, user-triggered, unnamed.
The row that surprised me: the SEO-tool trio (1,274 knocks) out-crawled every search engine combined (885). Majestic's MJ12bot alone out-fetched Googlebot's named count by roughly 47 to 1 — and MJ12 is named after the X-Files UFO office. Which of your crawlers would you actually miss if it stopped knocking?
@alexshev The audit-trail point stands. What we store per knock today is UA, path and timestamp — the shape the dashboard gives us without keeping header dumps of every visitor on a public path. Your robots-response column is the one I can add cheapest and it is the most diagnostic: it separates a bot that ignored a transient rule from one that obeyed, which is exactly the doubt you name in the last line. Next revision: per-knock status code plus ASN grouping, both offline, both in the same table.
On naming the anonymous 65.5%: stop looking at the user agent and look at who owns the IP. Group those 5,668 hits by ASN and organisation and most of them get a name fast, usually a handful of cloud and hosting networks plus a long tail of home ISPs. Reverse DNS plus the published crawler IP lists then confirms which of your 24 named bots are real and which are borrowing the name.
You can do the ASN step offline with a free IP-to-ASN dataset. If you'd rather call an API, IPGeolocation.io (IPGeolocation.io/ip-location-api.html) returns ASN and organisation on the free plan, 1,000 lookups a day (I work on it). The hosting and proxy flags are paid, but for a 7-night census the ASN alone does most of the work.
@furqan_ashraf Right diagnosis — the UA names the polite, the IP names the rest. Late follow-up (a rough week on our side): the next census cut groups the unnamed by ASN offline from the free IP-to-ASN dataset first — the hits stay ours. Your API works for spot checks, but a 7-day window wants the offline pass. The reverse-DNS confirmation also got backup below (the Googlebot check): that pass is how we prove a borrowed name — our Bingbot count jumped the day the crawler returned named, and the ASN check is what tells us it was never a spoof.
Makes sense, keeping the hits local is the right call for a full week. One tip for the offline pass: sort the IP-to-ASN ranges once and binary-search each IP against them instead of looping per line, and the 7 nights will run in seconds. Looking forward to the next census, happy to help if anything in the ASN grouping looks odd.
Since you've kept the user-agent strings, could you rerun all seven nights through the updated detection list? That would give Googlebot and the other newly named bots a full week's count too. It would also show how much of the anonymous 65% was a gap in your list rather than crawlers withholding their identity.
The finding I keep coming back to is that 65% of knocks had no bot name at all. It makes "block or allow AI crawlers in robots.txt" feel like a smaller lever than people assume, since the unnamed traffic won't read it anyway. Did you try grouping that unnamed share by ASN or IP range to see how much of it is the same crawlers in disguise?
The nine-to-one PerplexityBot vs OAI-SearchBot gap is interesting. One thing worth adding before reading it as interest: ChatGPT-User only shows up when a person asks ChatGPT about a page, so counting it next to OAI-SearchBot gives a fuller picture of how often ChatGPT actually reads the site. Did it appear at all in the seven nights?
The search crawler bargain was an exchange of bandwidth for referral traffic. You covered the server ingress and egress, and Googlebot sent human visits that converted into readers or subscribers. That trade had positive expected value for the publisher.
AI crawlers break that payoff structure. When an answering engine fetches a page to synthesize an answer directly on its own surface, referral probability drops toward zero while the hosting and compute costs remain on your ledger. The publisher underwrites the retrieval context for someone else's subscription revenue.
The 65 percent unnamed share is the predictable equilibrium. Once publishers realize the net payoff of named bot visits is negative and update robots.txt, crawlers face an immediate incentive to strip their user-agent headers and route through residential proxy pools. A voluntary identification standard cannot survive when the economics behind it turn purely extractive.
@deanlee This is the frame the whole thread has converged on — the referral column was the deal, and the answering engines fetch exactly the pages that used to earn it. A week of comments here has sharpened the identity side (ASN grouping, reverse-DNS); the column I still cannot print is referrals per bot. The honest-column fix is specced for the next census revision: knocks in one column, humans delivered in another, arithmetic labeled as arithmetic. Until reindex hands search traffic back, "bandwidth for nothing" is what the unnamed 65.5% charge us.
A short census becomes much more useful if each observation records request headers, IP or ASN, robots response, route, and timestamp. That separates crawler identity from traffic shape and makes later policy changes auditable. It also avoids drawing a broad conclusion from one bot that happened to ignore a transient rule.
The 8,654 bot knocks vs 247 human visits ratio is the kind of number I wish more robots.txt debates started with. The Perplexity-vs-OpenAI gap and the unnamed 65% are especially useful — did the unnamed share look like scrapers with rotating UAs, or mostly empty/generic agents?
Googlebot is the crawler I'd miss most, and the purpose column hides why: Google's AI Overviews and AI Mode answer from what plain Googlebot fetches, so those 14 knocks are search and AI at once.
Google has no separate AI user agent for a census to count. Google-Extended is a robots.txt token only; Google's crawler docs say it makes no requests of its own, and blocking it covers Gemini training and grounding, not AI Overviews. So splitting by user agent will undercount Google's AI demand however complete the detection list gets.
For the 14 itself, Search Console holds Google's side of the ledger: Settings, Crawl stats, which counts requests per day and splits them by Googlebot type. Since Googlebot was only named from Nov 1, compare that day's figure with yours. If Google's is far higher, the gap is your list, and the rerun Stefan asked for will close it. If they roughly match, Google is crawling you lightly, and that is the number worth watching, more than PerplexityBot's 273.
Two pushes on this one, both from the other side of the glass.
First: a chunk of your 65.5% unnamed pile probably isn't crawlers at all. On a small site, anonymous traffic is dominated by vulnerability scanners hammering /wp-login.php, /.env and /xmlrpc.php, plus credential-stuffing POSTs — they fetch one or two URLs, never follow links, never read content. Folding them into the same column as SemrushBot makes bot pressure look scarier than it is. A crude cut — unnamed requests that got 404/403, or asked for a non-page path — would likely shrink that 5,668 by half. My own noise floor agrees from an unexpected direction: the single biggest bot mess on my site turned out to be my own analytics plugin generating admin-ajax URLs, 46 of which robots.txt now blocks. Anonymous volume needs composition analysis before it counts as a census row.
Second: crawl volume is less a property of the bot than a verdict on the site. In Search Console, one of my sites had 30 URLs moved from indexed to "crawled - currently not indexed" in a single batch on Oct 16, and then Google re-crawled nothing for 15 days. Same robots.txt, same server, zero knocks — because it had already decided the pages weren't worth fetching. Your own table has the tell: Googlebot at 14 against Yandex's 643 isn't Yandex being nine times hungrier, it's Google fetching on a quality judgment while Yandex fetches on a schedule. A census of door knocks measures who still believes you're worth fetching; the decision happens before the knock.
Which makes the SEO-tool family (1,274 knocks) the really interesting row — those bots read your site to sell you ranking reports, and they don't stop when the quality verdict goes against you. The bots that knock most persistently are the ones monetizing your anxiety, not the ones sending you readers.
from the scraper side the user agent is the first thing anyone fakes, so the named count reads like a floor and not a share. whether a client asked for robots.txt before anything else probably splits that unnamed bucket better than the name does
The 65% unnamed bucket is the interesting part. Have you tried splitting it by ASN? In logs I've looked at, a big chunk of "no name" traffic comes from a handful of cloud/hosting ASNs, which is a decent hint it's scrapers rather than people. Also worth verifying the named ones by reverse DNS or published IP ranges, since spoofed Googlebot/GPTBot user agents are common. Googlebot at 14 hits feels low enough that I'd want to know whether some real Googlebot traffic landed in the unnamed bucket.
Reverse-DNS the IPs claiming to be Googlebot before you trust the name tag. A chunk of the "named" pile is spoofed the same way the unnamed two-thirds is — boring check, still catches a surprising number of fakes in access logs. Your Googlebot=14 footnote is the other useful bit: a migration that left redirects pointing at gone URLs will quietly starve the one crawler that actually sends traffic.