A large checklist becomes maintainable when each check has a decision attached: block release, create a ticket, or record a baseline. I’d also sample false positives and measure time-to-diagnosis, since a technically accurate test can still be a poor guardrail if nobody trusts or acts on it. The useful suite is the one that changes decisions.
Full-stack dev, 10+ years. I build accessible websites for Italian public institutions and open-source AI tools for developers. Founder of Auriti Labs 🇮🇹
You're right on the principle: each check needs a decision attached, or it's noise. The tier ranking in the article is that mapping. Tier 1 means fix now or stay invisible. Tier 3 means record a baseline, it doesn't predict anything. Tier 4 is structural, no citation prediction claimed.
On false positives and time-to-diagnosis: fair point the article doesn't cover. The calibration study tested the score at the suite level (came back null), but I haven't measured false positive rates for individual checks yet. For example, robots.txt detection has a false positive when a site uses a CDN that intercepts the crawl before robots.txt is served. Worth tracking.
One context difference: the tool is an audit, not a CI gate. It doesn't block deploys. But the principle holds. A check that doesn't change a decision is dead weight. The calibration study was exactly that. It showed the score doesn't predict citation, so the 0-100 number doesn't change decisions as much as the fix list does.
That audit framing makes the split even clearer: the dashboard should optimize for an ordered next action, while the calibration layer tells you how much to trust each check. Logging false positives by delivery path—origin, CDN, or WAF—would also turn those edge cases into routing rules rather than caveats buried in the report.
Full-stack dev, 10+ years. I build accessible websites for Italian public institutions and open-source AI tools for developers. Founder of Auriti Labs 🇮🇹
The tier ranking is already that ordered action list. Tier 1 fixes first, then Tier 2, down the line. The trust layer on top is what's missing.
Logging false positives by delivery path is a concrete improvement. Right now a robots.txt check treats a CDN interception and a real blocking rule the same way. Splitting them by path would let the tool say "probably CDN interception" instead of flagging both as failures. Not built yet, but it's where the check layer needs to go.
That distinction would make the trust layer actionable. I’d retain the observed evidence with each finding—request path, headers or cache signal, and where the response diverged—then expose a confidence label and a short remediation path. A CDN-shaped anomaly can then be triaged without pretending it proves a crawler block.
Full-stack dev, 10+ years. I build accessible websites for Italian public institutions and open-source AI tools for developers. Founder of Auriti Labs 🇮🇹
This is a solid data model for the finding layer. Retaining the request path, headers, and where the response diverged gives the confidence label something to stand on. A CDN interception and a real Disallow rule produce different evidence, and right now the tool treats both the same way.
One practical constraint: the tool is a stateless CLI. Capturing request/response metadata per finding means the check layer needs to collect that evidence during the audit, then carry it through to the output. That's a structural change to how checks work today. Feasible, but it belongs in the next rubric revision rather than a patch.
The confidence label is the easy part once the evidence is there. The hard part is collecting that evidence reliably across different CDN and WAF setups without introducing false negatives of its own.
In one scan this week, GPTBot and ClaudeBot got a bot-challenge page from a large site. Of the 30% of sites you found blocking at least one AI bot, how many used robots.txt and how many a WAF or CDN rule?
Full-stack dev, 10+ years. I build accessible websites for Italian public institutions and open-source AI tools for developers. Founder of Auriti Labs 🇮🇹
The tool checks the two mechanisms separately, but the full dataset only stores the robots.txt counts.
The audit reads robots.txt (Disallow rules per bot) and, in a separate step, requests the home page with the GPTBot, OAI-SearchBot, PerplexityBot and Claude-SearchBot user agents, comparing status code and body size with a normal browser request. Only the robots.txt counts are kept for the full 2,800-domain benchmark.
The HTTP check was stored for about 630 sites, mostly registered accounts and one-off audits. On those, 7% block at least one AI bot in robots.txt, and 9% return 403/429/451/503 to an AI user agent while the browser gets a 200. Counting challenge-page keywords and truncated bodies too, 16% look blocked at the HTTP level. Taking the strict definition: 31 sites block only through robots.txt, 45 only at the HTTP level, and 13 do both.
So on this subset, CDN/WAF-level blocking is at least as common as an explicit Disallow. Two caveats: the HTTP check has false positives (it doesn't follow redirects, and a page containing "blocked" or "captcha" can trip it), and it only sees status and body, so it can't distinguish Cloudflare rules from a Wordfence or Sucuri rule. About half of the HTTP flags are on Cloudflare-fronted sites.
I've also corrected the blocking rate in the article above. The original 30% figure was not supported by the benchmark data. The actual rate is 9.5% via robots.txt across 2,800 domains, rising to roughly 20% when HTTP-level blocking is included.
Full-stack dev, 10+ years. I build accessible websites for Italian public institutions and open-source AI tools for developers. Founder of Auriti Labs 🇮🇹
We can't tell from the data the tool collects today. It sees the outcome (status code, challenge page), not the Cloudflare configuration behind it.
What we know: Cloudflare Bot Fight Mode is off by default on new zones. If it's on, someone turned it on. But Cloudflare also has WAF managed rules and rate limiting that can intercept bots without the site owner explicitly configuring an AI crawler block.
One indirect signal: if the block hits only GPTBot or ClaudeBot specifically, it's more likely a configured rule. If it hits any unrecognized user agent, it's more likely Bot Fight Mode. The tool doesn't split those two cases in its output today, but it's a distinction that could be added by comparing which user agents get blocked on each site.
Our check asks for the page as GPTBot, as ClaudeBot and as a browser. A fourth request under a made-up agent name would sit next to those and show which case it is.
Would you count a block on the made-up name as Bot Fight Mode, or keep it as a third bucket?
Full-stack dev, 10+ years. I build accessible websites for Italian public institutions and open-source AI tools for developers. Founder of Auriti Labs 🇮🇹
Good idea. The fourth request is the cleanest way to split the two cases.
I wouldn't label it "Bot Fight Mode" specifically. From the outside we can't distinguish BFM from Super Bot Fight Mode, from a WAF custom rule blocking non-browser UAs, or from origin-level security plugins (Wordfence, Sucuri) that block unrecognized user agents. All of them produce the same observable result: the made-up UA gets blocked. The test tells us whether the block is generic or specific to AI crawlers, not which mechanism causes it.
Three buckets:
AI-specific block: AI bots blocked, generic UA passes
Generic bot block: everything blocked, generic UA included
Bot protection, AI allowed: generic UA blocked, AI bots pass
The third one is the case I care about most. It means the site has bot protection active but has chosen to let verified AI crawlers through. For the tool it's a pass on AI visibility, not a block, so it deserves its own bucket.
SEO monitoring tool. Crawls your site daily or hourly and alerts you when a noindex tag, robots.txt block, sitemap error or canonical change appears. Free plan.
Publishing a null calibration result and keeping the checks with honest weights is more than most tools in this space do. The CDN/WAF number stood out to me. A Disallow line is easy to spot in robots.txt, but a bot challenge at the edge never shows up there, so a site can look wide open and still hand GPTBot a 403. Are you seeing that more from default bot-protection settings, or from rules people added on purpose and forgot about?
We're a place where coders share, stay up-to-date and grow their careers.
A large checklist becomes maintainable when each check has a decision attached: block release, create a ticket, or record a baseline. I’d also sample false positives and measure time-to-diagnosis, since a technically accurate test can still be a poor guardrail if nobody trusts or acts on it. The useful suite is the one that changes decisions.
You're right on the principle: each check needs a decision attached, or it's noise. The tier ranking in the article is that mapping. Tier 1 means fix now or stay invisible. Tier 3 means record a baseline, it doesn't predict anything. Tier 4 is structural, no citation prediction claimed.
On false positives and time-to-diagnosis: fair point the article doesn't cover. The calibration study tested the score at the suite level (came back null), but I haven't measured false positive rates for individual checks yet. For example, robots.txt detection has a false positive when a site uses a CDN that intercepts the crawl before robots.txt is served. Worth tracking.
One context difference: the tool is an audit, not a CI gate. It doesn't block deploys. But the principle holds. A check that doesn't change a decision is dead weight. The calibration study was exactly that. It showed the score doesn't predict citation, so the 0-100 number doesn't change decisions as much as the fix list does.
That audit framing makes the split even clearer: the dashboard should optimize for an ordered next action, while the calibration layer tells you how much to trust each check. Logging false positives by delivery path—origin, CDN, or WAF—would also turn those edge cases into routing rules rather than caveats buried in the report.
The tier ranking is already that ordered action list. Tier 1 fixes first, then Tier 2, down the line. The trust layer on top is what's missing.
Logging false positives by delivery path is a concrete improvement. Right now a robots.txt check treats a CDN interception and a real blocking rule the same way. Splitting them by path would let the tool say "probably CDN interception" instead of flagging both as failures. Not built yet, but it's where the check layer needs to go.
That distinction would make the trust layer actionable. I’d retain the observed evidence with each finding—request path, headers or cache signal, and where the response diverged—then expose a confidence label and a short remediation path. A CDN-shaped anomaly can then be triaged without pretending it proves a crawler block.
This is a solid data model for the finding layer. Retaining the request path, headers, and where the response diverged gives the confidence label something to stand on. A CDN interception and a real Disallow rule produce different evidence, and right now the tool treats both the same way.
One practical constraint: the tool is a stateless CLI. Capturing request/response metadata per finding means the check layer needs to collect that evidence during the audit, then carry it through to the output. That's a structural change to how checks work today. Feasible, but it belongs in the next rubric revision rather than a patch.
The confidence label is the easy part once the evidence is there. The hard part is collecting that evidence reliably across different CDN and WAF setups without introducing false negatives of its own.
In one scan this week, GPTBot and ClaudeBot got a bot-challenge page from a large site. Of the 30% of sites you found blocking at least one AI bot, how many used robots.txt and how many a WAF or CDN rule?
The tool checks the two mechanisms separately, but the full dataset only stores the robots.txt counts.
The audit reads robots.txt (Disallow rules per bot) and, in a separate step, requests the home page with the GPTBot, OAI-SearchBot, PerplexityBot and Claude-SearchBot user agents, comparing status code and body size with a normal browser request. Only the robots.txt counts are kept for the full 2,800-domain benchmark.
The HTTP check was stored for about 630 sites, mostly registered accounts and one-off audits. On those, 7% block at least one AI bot in robots.txt, and 9% return 403/429/451/503 to an AI user agent while the browser gets a 200. Counting challenge-page keywords and truncated bodies too, 16% look blocked at the HTTP level. Taking the strict definition: 31 sites block only through robots.txt, 45 only at the HTTP level, and 13 do both.
So on this subset, CDN/WAF-level blocking is at least as common as an explicit Disallow. Two caveats: the HTTP check has false positives (it doesn't follow redirects, and a page containing "blocked" or "captcha" can trip it), and it only sees status and body, so it can't distinguish Cloudflare rules from a Wordfence or Sucuri rule. About half of the HTTP flags are on Cloudflare-fronted sites.
I've also corrected the blocking rate in the article above. The original 30% figure was not supported by the benchmark data. The actual rate is 9.5% via robots.txt across 2,800 domains, rising to roughly 20% when HTTP-level blocking is included.
We run a check for that gap. It asks for the page as an AI agent and as a browser, then compares what comes back.
On the Cloudflare half, did the block look like a default for new zones, or a switch the owner turned on?
We can't tell from the data the tool collects today. It sees the outcome (status code, challenge page), not the Cloudflare configuration behind it.
What we know: Cloudflare Bot Fight Mode is off by default on new zones. If it's on, someone turned it on. But Cloudflare also has WAF managed rules and rate limiting that can intercept bots without the site owner explicitly configuring an AI crawler block.
One indirect signal: if the block hits only GPTBot or ClaudeBot specifically, it's more likely a configured rule. If it hits any unrecognized user agent, it's more likely Bot Fight Mode. The tool doesn't split those two cases in its output today, but it's a distinction that could be added by comparing which user agents get blocked on each site.
Our check asks for the page as GPTBot, as ClaudeBot and as a browser. A fourth request under a made-up agent name would sit next to those and show which case it is.
Would you count a block on the made-up name as Bot Fight Mode, or keep it as a third bucket?
Good idea. The fourth request is the cleanest way to split the two cases.
I wouldn't label it "Bot Fight Mode" specifically. From the outside we can't distinguish BFM from Super Bot Fight Mode, from a WAF custom rule blocking non-browser UAs, or from origin-level security plugins (Wordfence, Sucuri) that block unrecognized user agents. All of them produce the same observable result: the made-up UA gets blocked. The test tells us whether the block is generic or specific to AI crawlers, not which mechanism causes it.
Three buckets:
The third one is the case I care about most. It means the site has bot protection active but has chosen to let verified AI crawlers through. For the tool it's a pass on AI visibility, not a block, so it deserves its own bucket.
Publishing a null calibration result and keeping the checks with honest weights is more than most tools in this space do. The CDN/WAF number stood out to me. A Disallow line is easy to spot in robots.txt, but a bot challenge at the edge never shows up there, so a site can look wide open and still hand GPTBot a 403. Are you seeing that more from default bot-protection settings, or from rules people added on purpose and forgot about?