DEV Community

Cover image for Agentic Pentest Frameworks (2026): The Full Field Guide

Agentic Pentest Frameworks (2026): The Full Field Guide

Mikalai Karytka on October 28, 2026

A few days ago I published a short map of the LLM-driven pentest landscape. I promised a detailed follow-up. This is it. Full comparison table, a t...
Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The scope-enforcement section is the part most surveys skip — star counts tell you nothing, but 'prompt-only' versus 'code pre-call' tells you what happens the moment an agent goes sideways. The open ground you name, an allow-list plus egress lockdown, is the real takeaway: a sandbox contains the blast radius but does not stop a confused agent burning its budget inside the allowed box. In your survey, did any framework do per-call scope checks that track the target itself, like blocking a new host mid-run, rather than just the command?

Collapse
 
mkarytka profile image
Mikalai Karytka •

Great question, and it's the one that matters. Short answer from the survey: no.

The one framework doing an in-code pre-call check (AI-OPS) gates the command, not the target. It asks "is this tool allowed to fire," not "is this host still in scope." Nothing I looked at does a per-call, target-aware check that would catch a new host appearing mid-run and block it.
The closest primitive is RedAmon's socket-proxy broker, which at least brokers connections rather than commands. But that, and the egress locks in Strix/PentAGI/Nebula, still contain at the network layer: they bound what the box can reach, they don't assert "this specific new host is out of scope, refuse" on the call. Containment, not a scope check, which is exactly the line you drew.
This bit me first-hand. The agents follow redirects, including cross-host 302s, without treating the hop as a risk at all. What actually refused an out-of-scope redirect was the egress policy I'd agreed up front, plus the provider-side safety ("not a verified security research engagement"). Neither is the tool asserting scope. The tool was happy to keep going. So target-aware per-call scope is still open ground, same as the in-code allow-list.
On your budget point, that's a second gap worth naming on its own. Egress bounds where it can go, not how much it can burn inside the box. The only brakes I saw on an agent thrashing within scope were coarse step or time caps and a human watching. "You're in bounds but clearly lost, stop" doesn't exist yet either.
Token economy was never a goal of this experiment, so I have no hard numbers there. The aim was preparatory lab work kept as close to a real engagement as I could get, not cost optimization. And honestly, work like this really should run on a controlled local model in a well-controlled environment to begin with, where you can watch the agent, cap it, and reproduce a run without a hosted model's budget and safety behavior sitting in the middle of what you're trying to measure.

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The redirect example is the sharpest evidence in the whole answer — a cross-host 302 is a new target decision the agent never made, yet nothing treats it as one. That also reframes where the check belongs: if the hop itself required re-authorization against scope, the redirect would be caught without a whole new layer. And the 'in bounds but clearly lost' gap pairs with the budget point — a step cap counts motion, not progress, so would a goal-progress check (are we closer to the objective than N steps ago) be the brake that actually fits, or does that just move the judgment call into the checker?

Thread Thread
 
mkarytka profile image
Mikalai Karytka •

Caveat first, I ran these tools, I didn't build them, so this is a user's take more than a design one.
The hop-reauth idea is the one I'd put money on, mainly because it needs no judgment. Is this host in the authorized set, yes or no, deny by default. That's something a tester can actually verify. The progress-check side is more interesting, but it's past what I can tell you from just running things. My gut's the same as yours: it shoves the hard call into the checker, and progress in this work isn't steady enough for "closer than N steps ago" to mean much. That's me guessing, not measuring.
What I'd stand behind from running them is that the dumb signals are the ones I trust. Step and time caps, plus catching when a run stops turning up anything new. Whether a smarter brake is worth building is a question for whoever builds it. If that's you, I'd read whatever you come up with.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The dumb signals earn trust precisely because they're verifiable — a host either is or isn't in the authorized set, and a cap either fired or it didn't. One definition I'd still want pinned down: what counts as 'stops turning up anything new' — no new hosts, no new findings, or no new anything for N steps?

Thread Thread
 
mkarytka profile image
Mikalai Karytka •

"No new findings" is what you care about but the worst brake: findings are sparse and bursty, and only count once validated, so you'd kill runs right before the payoff. "No new hosts" is too blunt, since real progress often happens deep in one host, an auth flow or param fuzzing, with no new host in sight.
The definition I'd use is at the action level: have I seen this (method, normalized URL, param set, response class) before, yes or no. A new endpoint, param, status class or auth state counts. Re-issuing a request you've already made doesn't. Same checkable test as host-in-set, just on the action.
And I'd make it a flag, not a kill: "no new surface in N steps, come look," not "terminate."
One more thing worth adding: the input prompt moves this more than any brake does. Same tools, same target, everything else equal, and the way you phrase the prompt decides how deep the scan goes and whether a given bug even surfaces. That's why none of these questions has a single right answer, there are just too many moving parts to pin one down. Good topic to chew on though. :)