I write about AI automation and side hustles for solo operators at SoloAgentFlow. n8n, Make and Claude workflows, case studies, and honest notes on what did not work.
The blocked-but-empty case is nasty because a block page and a genuinely empty search look identical to the agent. Two cheap guards worth trying: log result count (or raw page size) per source per round and alert when a source that normally returns something hits zero a few rounds in a row, and add one known-good canary query each round. If the canary comes back empty, mark the round as blocked instead of no results, so it never gets counted as a clean zero.
One man. An army of AI agents. I build AI workers that find leads, answer customers and run ops 24/7 for real businesses. Also building TommyAPI: one key, every top model.
The canary query is a good one. In that run it was 5 empty rounds out of 13 before anyone looked, and a known-good query each round would have flagged the block on round one. Worth stealing.
the 5 empty rounds are the scary part bc the final number still looked fine. travel has the same shape, a hotel sitting in an itinerary looks booked until someone checks it actually is, its why searchspot keeps shortlisted and confirmed as separate things
One man. An army of AI agents. I build AI workers that find leads, answer customers and run ops 24/7 for real businesses. Also building TommyAPI: one key, every top model.
Yes, that is the part that bugged me most. The day still ended with 8 leads, which looked like a normal number, so nobody asked why 5 of the 13 rounds came back empty. Shortlisted vs confirmed is the same split I was missing: the round ran vs the round actually returned something.
An empty result is a valid return for most tools, so the agent accepts it and moves on. Check the shape at the tool boundary: a count that must be above zero, a required field that must be present. The empty answer then fails on the first call.
One man. An army of AI agents. I build AI workers that find leads, answer customers and run ops 24/7 for real businesses. Also building TommyAPI: one key, every top model.
Agreed. By its own contract the tool did nothing wrong, empty was a legal answer. The check has to sit where the result comes back. I hit a cousin of this too: a scheduled job reported success every single time while the script never actually started, and an alert channel sat silent from one afternoon to the next day.
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. kevin@outricher.com
the silent zero is why i like tools that return a row saying what happened instead of nothing, a skipped person comes back with a status like no email or over the cap and the agent reads the count of those first. one run of that kind with the inputs is here dev.to/kevinhall/cheap-clay-and-ap...
One man. An army of AI agents. I build AI workers that find leads, answer customers and run ops 24/7 for real businesses. Also building TommyAPI: one key, every top model.
A status row per skipped item beats a bare count, you can see the why at a glance. Empty and blocked looked identical to my agent because neither one left anything behind. Even a single line saying "blocked, page looked like a bot check" would have saved me 4 rounds.
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. kevin@outricher.com
One man. An army of AI agents. I build AI workers that find leads, answer customers and run ops 24/7 for real businesses. Also building TommyAPI: one key, every top model.
Fair point, I undersold the count. The line explains one run, the tally shows which reason is creeping up. My 5 empty rounds out of 13 each looked fine on their own, and the day still ended with 8 leads. A count per reason printed every round would have made the blocked case visible long before the day was over.
This highlights one of the most important and underestimated challenges in production AI systems: a successful execution does not necessarily mean a successful outcome. The example of five consecutive empty runs is particularly revealing because the agent wasn't technically malfunctioning; it was operating under misleading environmental conditions and reporting those conditions as legitimate results. That's arguably more dangerous than an exception because conventional monitoring has nothing obvious to flag. I especially appreciate the distinction between checking whether a process is alive and verifying whether it can actually complete the task it exists to perform. This is the difference between infrastructure monitoring and outcome-based observability. The point about measuring cost per finished task is equally important. Token efficiency and low execution costs are meaningless if an agent repeatedly fails to deliver usable results. I'd also argue that confidence thresholds, evidence freshness, and explicit classifications for empty, incomplete, and verified results should become standard parts of agent reporting. The retired-rule example exposes another subtle problem: agent memory and persistent instructions need lifecycle management just like production code, because outdated context can become an invisible source of behavioral failure. Ultimately, dependable agents need more than exception handling; they need mechanisms that question suspiciously successful executions, validate real-world outcomes, and make uncertainty visible. The best monitoring system isn't the one that tells us everything is green. It's the one that tells us when green doesn't actually mean working. Excellent practical lessons for anyone moving AI agents beyond experimentation into production.
One man. An army of AI agents. I build AI workers that find leads, answer customers and run ops 24/7 for real businesses. Also building TommyAPI: one key, every top model.
Agreed on explicit labels for empty vs incomplete vs verified. The one that cost me most was a scheduled job that showed success on every run while the script never actually started, so a lead alert channel sat idle from one afternoon to the next day. Now every run has to write a log line, and no log line counts as not done.
@tommybuilds same trap here: a test that didn't compile wrote no FAILED line, so the run read as green. Counting what a run must produce catches it; the exit code doesn't.
One man. An army of AI agents. I build AI workers that find leads, answer customers and run ops 24/7 for real businesses. Also building TommyAPI: one key, every top model.
The compile one is nasty because nothing even ran, so nothing could fail. I hit a cousin of it today: a small model sorting 20 customer messages got 9 wrong, and on 6 of those it reported full confidence. The score looked as healthy as your green run. Now I check the count first (20 in, 20 labelled) before I look at any score.
@tommybuilds same order here: the count comes before the score. If the labelled rows don't match the inputs, the score is a number about whichever rows happened to get a label.
One man. An army of AI agents. I build AI workers that find leads, answer customers and run ops 24/7 for real businesses. Also building TommyAPI: one key, every top model.
Right, and a score over a partial set still looks precise, which is what makes it dangerous. Printing in vs labelled on the line above the score is the cheapest fix I know of.
We're a place where coders share, stay up-to-date and grow their careers.
The blocked-but-empty case is nasty because a block page and a genuinely empty search look identical to the agent. Two cheap guards worth trying: log result count (or raw page size) per source per round and alert when a source that normally returns something hits zero a few rounds in a row, and add one known-good canary query each round. If the canary comes back empty, mark the round as blocked instead of no results, so it never gets counted as a clean zero.
The canary query is a good one. In that run it was 5 empty rounds out of 13 before anyone looked, and a known-good query each round would have flagged the block on round one. Worth stealing.
the 5 empty rounds are the scary part bc the final number still looked fine. travel has the same shape, a hotel sitting in an itinerary looks booked until someone checks it actually is, its why searchspot keeps shortlisted and confirmed as separate things
Yes, that is the part that bugged me most. The day still ended with 8 leads, which looked like a normal number, so nobody asked why 5 of the 13 rounds came back empty. Shortlisted vs confirmed is the same split I was missing: the round ran vs the round actually returned something.
An empty result is a valid return for most tools, so the agent accepts it and moves on. Check the shape at the tool boundary: a count that must be above zero, a required field that must be present. The empty answer then fails on the first call.
Agreed. By its own contract the tool did nothing wrong, empty was a legal answer. The check has to sit where the result comes back. I hit a cousin of this too: a scheduled job reported success every single time while the script never actually started, and an alert channel sat silent from one afternoon to the next day.
the silent zero is why i like tools that return a row saying what happened instead of nothing, a skipped person comes back with a status like no email or over the cap and the agent reads the count of those first. one run of that kind with the inputs is here dev.to/kevinhall/cheap-clay-and-ap...
A status row per skipped item beats a bare count, you can see the why at a glance. Empty and blocked looked identical to my agent because neither one left anything behind. Even a single line saying "blocked, page looked like a bot check" would have saved me 4 rounds.
the reason line is the cheap part, and a count per reason on every run shows when one of them starts to grow
Fair point, I undersold the count. The line explains one run, the tally shows which reason is creeping up. My 5 empty rounds out of 13 each looked fine on their own, and the day still ended with 8 leads. A count per reason printed every round would have made the blocked case visible long before the day was over.
This highlights one of the most important and underestimated challenges in production AI systems: a successful execution does not necessarily mean a successful outcome. The example of five consecutive empty runs is particularly revealing because the agent wasn't technically malfunctioning; it was operating under misleading environmental conditions and reporting those conditions as legitimate results. That's arguably more dangerous than an exception because conventional monitoring has nothing obvious to flag. I especially appreciate the distinction between checking whether a process is alive and verifying whether it can actually complete the task it exists to perform. This is the difference between infrastructure monitoring and outcome-based observability. The point about measuring cost per finished task is equally important. Token efficiency and low execution costs are meaningless if an agent repeatedly fails to deliver usable results. I'd also argue that confidence thresholds, evidence freshness, and explicit classifications for empty, incomplete, and verified results should become standard parts of agent reporting. The retired-rule example exposes another subtle problem: agent memory and persistent instructions need lifecycle management just like production code, because outdated context can become an invisible source of behavioral failure. Ultimately, dependable agents need more than exception handling; they need mechanisms that question suspiciously successful executions, validate real-world outcomes, and make uncertainty visible. The best monitoring system isn't the one that tells us everything is green. It's the one that tells us when green doesn't actually mean working. Excellent practical lessons for anyone moving AI agents beyond experimentation into production.
Agreed on explicit labels for empty vs incomplete vs verified. The one that cost me most was a scheduled job that showed success on every run while the script never actually started, so a lead alert channel sat idle from one afternoon to the next day. Now every run has to write a log line, and no log line counts as not done.
@tommybuilds same trap here: a test that didn't compile wrote no FAILED line, so the run read as green. Counting what a run must produce catches it; the exit code doesn't.
The compile one is nasty because nothing even ran, so nothing could fail. I hit a cousin of it today: a small model sorting 20 customer messages got 9 wrong, and on 6 of those it reported full confidence. The score looked as healthy as your green run. Now I check the count first (20 in, 20 labelled) before I look at any score.
@tommybuilds same order here: the count comes before the score. If the labelled rows don't match the inputs, the score is a number about whichever rows happened to get a label.
Right, and a score over a partial set still looks precise, which is what makes it dangerous. Printing in vs labelled on the line above the score is the cheapest fix I know of.