DEV Community

Cover image for Agentic Pentest Frameworks (2026): The Full Field Guide
Mikalai Karytka
Mikalai Karytka

Posted on Edited on

Agentic Pentest Frameworks (2026): The Full Field Guide

A few days ago I published a short map of the LLM-driven pentest landscape. I promised a detailed follow-up. This is it. Full comparison table, a taxonomy that makes the field make sense, deep profiles of all twelve projects, quick-start commands for the top three, and where Claude fits in each.

It's a tooling survey built from public repos and docs. Nothing here is an attack guide, and nothing is tied to a specific target. Stars and last-push dates are a snapshot from 2026-10-26, verified via the GitHub API; treat them as a moment in time.

TL;DR: Sort the field on two axes (where the reasoning lives, how much product wraps it) and twelve repos collapse into four buckets. For most work it's one of three: Strix (fastest to a running autonomous agent), HexStrike (the widest tool arsenal, you supply the brain), or PentAGI (a standing platform with memory and a UI). Shannon is the white-box specialist; RedAmon closes the loop with an auto-fix PR. The real bottleneck everywhere is tool-calling discipline, and a strong model removes it.

The mistake everyone makes

People line these projects up and compare star counts. That's comparing a library to a product to a protocol server. The field only makes sense once you place each tool on two axes:

  • Where does the reasoning live? External client ↔ built-in.
  • How much product/infrastructure wraps it? Library ↔ full platform.

Taxonomy: where the brain lives vs how much product wraps it

That gives four categories:

  1. Tool layers: hands, no brain. Reasoning is an external client/LLM; no agent loop and no built-in finding validation. → HexStrike AI, Reaper.
  2. Builder frameworks: SDKs to assemble your own agent. They ship example use-cases, but the logic is yours. → hackingBuddyGPT, CAI (archived).
  3. Autonomous agents: the loop (reason → tool-call → observe → decide) ships in the box; many self-validate findings. → Strix, Shannon, PentestGPT, Pentest Copilot, Nebula, AI-OPS.
  4. Platforms: an autonomous agent plus a product: web UI, persistent memory, Automation + human-in-the-loop, observability. → PentAGI, RedAmon.

Boundaries blur (RedAmon is a pipeline shipped as a platform; Pentest Copilot is an agent that also exposes a tool layer over MCP), but the four buckets are the right mental model.

The full comparison

Maintenance: 🟢 active (push < 3 mo) · 🟡 slowing · 🔴 archived. Snapshot 2026-10-26.

Project ★ Push Status License Type Architecture (short) Claude
Strix 72.0k 10-25 🟢 Apache-2.0 autonomous multi-agent Graph of Agents (OpenAI Agents SDK), every turn = tool call ✅ first-class
PentAGI 32.5k 10-22 🟢 MIT platform + multi-agent orchestrator → specialists, Postgres+pgvector, opt. Neo4j ✅ env/WIF, per-agent
HexStrike 13.1k 08-03 🟢 MIT MCP tool layer FastMCP server (:8888) + bridge, brain external ✅ main path
Shannon 49.4k 10-15 🟢 AGPL-3.0 autonomous agent (web/API) white-box: source analysis + browser/CLI → PoC ✅ provider-agnostic
RedAmon 3.2k 10-22 🟢 MIT autonomous pipeline recon→exploit→post-ex→triage→codefix→PR, Neo4j, 7 containers ✅ via harness
PentestGPT 16.1k 07-14 🟢 MIT agent + assistant multi-stage pipeline, drives Claude Code/Codex ✅ backend
Pentest Copilot 1.5k 08-14 🟢 MIT agent (browser) loop ≤25 iters, 16 tools, Burp, subagents, exposes MCP ✅ provider/MCP
Nebula 1.1k 10-25 🟢 BSD-2 harness + agents Butler → agents → subagents, SQLite+Chroma memory ✅ connect
AI-OPS 168 10-25 🟢 MIT agent behind HTTP API API container + CLI, litellm, for mid-size LLMs ✅ via litellm
hackingBuddyGPT 1.3k 10-24 🟢 MIT builder framework use-cases, planner/executor, SSH/shell connectors, RAG ✅ via provider
Reaper 0.9k 03-24 🟡 Apache-2.0 MITM proxy (AppSec) traffic capture to DB + CLI, agent-friendly ⚠️ via skills
CAI 9.8k 08-22 🔴 archived builder framework agent handoffs/tracing, 300+ backends (LiteLLM) ✅ (archived)

Popularity vs maintenance: stars (log) against last commit

Read that chart as a story, not a ranking. The most-starred projects (Strix, Shannon, PentAGI) are also the most actively maintained; top-right is where you want to be. CAI sits at ~10k stars but is archived (gray): popularity is not maintenance, and an unmaintained offensive framework is a liability, not a win.

Where scope is enforced

One axis the big table hides: when an agent decides a target is in scope, is that enforced in the prompt, checked in code before each tool call, or backstopped by the sandbox it runs in? It's easy to overlook when a run goes well. It matters the moment one goes sideways, a prompt-injected page or a confused agent reaching for something out of bounds, which is exactly when a prompt-level rule folds.

Three mechanisms, weakest to strongest:

  • Prompt only. The instructions say "stay in scope" and the model's judgment is the single gate. HexStrike (brain is external, no built-in guard), Pentest Copilot (commands run straight on the attack box), and the builder kits (hackingBuddyGPT, CAI, where scope is whatever you code) live here.
  • Sandbox / egress. Scope becomes "what the box can reach." Strix, PentAGI, Shannon (repo mounted read-only), RedAmon (a socket-proxy broker), and Nebula (optional OpenVPN egress) contain the blast radius even when the agent misjudges. Strong backstop, but it's containment, not a per-call scope check.
  • Code, pre-call. A command allow-list checked in code before the tool fires, default-deny. AI-OPS is the clearest example, allow-list-guarded by design precisely because it targets modest local models and can't trust the model to police itself.

The gap worth noticing: almost nobody does an explicit in-code scope check and a sandbox. Most pick one, and much of the field still leans on the prompt alone. A pre-call allow-list plus egress lockdown is largely open ground if you're building in this space. (Thanks to a reader building exactly that for the nudge to spell this out.)

The three you'll actually reach for

Strix: autonomous agent, batteries included

GitHub logo usestrix / strix

Open-source AI penetration testing tool to find and fix your app’s vulnerabilities.

Strix Banner

Strix

The open-source AI pentesting tool. Autonomous AI hackers that find and fix your app’s vulnerabilities.

Docs Website

Strix Cloud Try Strix Enterprise

GitHub Stars License PyPI Version

Join Discord Follow on X

usestrix%2Fstrix | Trendshift usestrix/strix | Trendshift

Tip

New! Strix integrates seamlessly with GitHub Actions and CI/CD pipelines. Automatically scan for vulnerabilities on every pull request and block insecure code before it reaches production - Get started with no setup required.


Strix Overview

Strix are autonomous AI penetration testing agents that act just like real hackers - they run your code dynamically, find vulnerabilities, and validate them through actual proofs-of-concept. Built for developers and security teams who need fast, accurate security testing without the overhead of manual pentesting or the false positives of static analysis tools.

Key Capabilities:

  • Full pentesting toolkit - reconnaissance, exploitation, and validation out of the box
  • Multi-agent orchestration - teams of AI pentesters that collaborate and scale
  • Real exploit validation - working PoCs, not false positives like legacy vulnerability scanners
  • Developer‑first CLI - actionable findings with…

A Graph of Agents on the OpenAI Agents SDK: a root agent delegates to sub-agents (recon, injection, auth…) via create_agent, and every turn must end in a tool call, which keeps reasoning from drifting into prose. Findings are validated with working PoCs before they hit the report. Execution runs in a sandbox. It's mainly an MCP client, though a hosted service also exposes an MCP server over its API.

Tools: HTTP intercept (Caido), browser testing, shell, a Python runtime for exploits, recon/OSINT, SAST+DAST, a findings KB with CVSS/OWASP tags.
Pros: ready agent; PoC validation + auto-fix + reports; CI mode; LiteLLM routing; minimal infra.
Cons: smaller arsenal than HexStrike; quality tracks the model hard; no rich UI/HITL.

# representative: check the repo README for current flags
pipx install strix-agent
export STRIX_LLM="anthropic/claude-sonnet-4-x"
export LLM_API_KEY="sk-ant-..."
strix --target https://your-authorized-target.example
Enter fullscreen mode Exit fullscreen mode

HexStrike AI: the arsenal, brain sold separately

GitHub logo 0x4m4 / hexstrike-ai

HexStrike AI MCP Agents is an advanced MCP server that lets AI agents (Claude, GPT, Copilot, etc.) autonomously run 150+ cybersecurity tools for automated pentesting, vulnerability discovery, bug bounty automation, and security research. Seamlessly bridge LLMs with real-world offensive security capabilities.

HexStrike AI Logo

HexStrike AI MCP Agents v6.0

AI-Powered MCP Cybersecurity Automation Platform

Python License Security MCP Version Tools Agents Stars

Advanced AI-powered penetration testing MCP framework with 150+ security tools and 12+ autonomous AI agents

Owned & developed by OTT Cybersecurity LLC

📋 What's New • 🏗️ Architecture • 🚀 Installation • 🛠️ Features • 🤖 AI Agents • 📡 API Reference


Follow Our Social Accounts

Join our Discord    Follow us on LinkedIn



Architecture Overview

HexStrike AI MCP v6.0 features a multi-agent architecture with autonomous AI agents, intelligent decision-making, and vulnerability intelligence.


<div class="js-render-enrichment-target" data-json='{"data":"%%{init: {\"themeVariables\": {\n \"primaryColor\": \"#b71c1c\",\n \"secondaryColor\": \"#ff5252\",\n \"tertiaryColor\": \"#ff8a80\",\n \"background\": \"#2d0000\",\n \"edgeLabelBackground\":\"#b71c1c\",\n \"fontFamily\": \"monospace\",\n \"fontSize\": \"16px\",\n \"fontColor\": \"#fffde7\",\n \"nodeTextColor\": \"#fffde7\"\n}}}%%\ngraph TD\n A[AI Agent - Claude/GPT/Copilot] -->|MCP Protocol| B[HexStrike MCP Server v6.0]\n \n B --> C[Intelligent Decision Engine]\n B --> D[12+ Autonomous AI Agents]\n B --> E[Modern Visual Engine]\n \n C --> F[Tool Selection AI]\n C --> G[Parameter Optimization]\n C --> H[Attack Chain Discovery]\n \n D --> I[BugBounty Agent]\n D --> J[CTF Solver Agent]\n D --> K[CVE Intelligence Agent]\n D --> L[Exploit Generator Agent]\n \n E --> M[Real-time Dashboards]\n E --> N[Progress Visualization]\n E --> O[Vulnerability Cards]\n \n B --> P[150+ Security Tools]\n P --> Q[Network Tools - 25+]\n P --> R[Web App Tools - 40+]\n P --> S[Cloud Tools - 20+]\n P --> T[Binary Tools - 25+]\n P --> U[CTF Tools - 20+]\n P --> V[OSINT Tools - 20+]\n \n B --> W[Advanced Process Management]\n W --> X[Smart Caching]\n W --> Y[Resource Optimization]\n W --> Z[Error Recovery]\n \n style A fill:#b71c1c,stroke:#ff5252,stroke-width:3px,color:#fffde7\n style B fill:#ff5252,stroke:#b71c1c,stroke-width:4px,color:#fffde7\n style C fill:#ff8a80,stroke:#b71c1c,stroke-width:2px,color:#fffde7\n style D fill:#ff8a80,stroke:#b71c1c,stroke-width:2px,color:#fffde7\n style E fill:#ff8a80,stroke:#b71c1c,stroke-width:2px,color:#fffde7\n"}' data-plain='%%{init: {"themeVariables": {
"primaryColor": "#b71c1c"
"secondaryColor": "#ff5252",
"tertiaryColor": "#ff8a80",
"background": "#2d0000",
"edgeLabelBackground":"#b71c1c",
"fontFamily": "monospace",
"fontSize": "16px",
"fontColor": "#fffde7",
"nodeTextColor": "#fffde7"
}}}%%
graph TD
A[AI Agent - Claude/GPT/Copilot] -->|MCP Protocol| B[HexStrike MCP Server v6.0]

B --&gt; C[Intelligent Decision Engine]
B --&gt; D[12+ Autonomous AI Agents]
B --&gt; E[Modern Visual Engine]

C --&gt; F[Tool Selection AI]
C --&gt; G[Parameter Optimization]
C --&gt; H[Attack Chain Discovery]

D --&gt; I[BugBounty Agent]
D --&gt; J[CTF Solver Agent]
D --&gt; K[CVE Intelligence Agent]
D --&gt; L[Exploit Generator Agent]

E --&gt; M[Real-time Dashboards]
E --&gt; N[Progress Visualization]
E --&gt; O[Vulnerability Cards]

B --&gt; P[150+ Security Tools]
P --&gt; Q[Network Tools - 25+]
P --&gt; R[Web App Tools - 40+]
P --&gt; S[Cloud Tools - 20+]
P --&gt; T[Binary Tools - 25+]
P --&gt; U[CTF Tools - 20+]
P --&gt; V[OSINT Tools - 20+]

B --&gt; W[Advanced Process Management]
W --&gt; X[Smart Caching]
W --&gt; Y[Resource Optimization]
W --&gt; Z[Error Recovery]

style A fill:#b71c1c,stroke:#ff5252,stroke-width:3px,color:#fffde7
style B fill:#ff5252,stroke:#b71c1c,stroke-width:4px,color:#fffde7
style C fill:#ff8a80,stroke:#b71c1c,stroke-width:2px,color:#fffde7
style D fill:#ff8a80,stroke:#b71c1c,stroke-width:2px,color:#fffde7
style E fill:#ff8a80,stroke:#b71c1c,stroke-width:2px,color:#fffde7
Enter fullscreen mode Exit fullscreen mode

' dir="auto">


%%{init: {"themeVariables": {
"primaryColor": "#b71c1c"
"secondaryColor": "#ff5252"
"tertiaryColor": "#ff8a80",
"background": "#2d0000",
"edgeLabelBackground":"#b71c1c",
"fontFamily": "monospace",
"fontSize": "16px",
"fontColor": "#fffde7",
"nodeTextColor": "#fffde7"
}}}%%
graph TD
A[AI Agent - Claude/GPT/Copilot] -->|MCP Protocol| B[HexStrike MCP Server v6.0]
B --&gt; C[Intelligent Decision Engine]
B --&gt; D[12+ Autonomous AI Agents]
B --&gt; E[Modern Visual Engine]

C --&gt; F[Tool Selection AI]
C --&gt; G[Parameter Optimization]
C --&gt; H[Attack Chain Discovery]

D --&gt; I[BugBounty Agent]
D --&gt; J[CTF Solver Agent]
D --&gt; K[CVE</pre>…</div>
Enter fullscreen mode Exit fullscreen mode




A server (hexstrike_server.py on :8888) wrapping its tools (150-plus by the project's own count) as MCP functions, plus a REST API (/api/intelligence/*, /api/command). Helper components exist (a decision engine, workflow managers), but the real reasoning is the attached MCP client. Tools: that 150-plus span Network/Web/Cloud/Binary/CTF/OSINT (nmap, nuclei, sqlmap, Ghidra, Prowler, Trivy…).
Pros: widest ready arsenal; clean brain/hands split; REST for custom pipelines.
Cons: not self-sufficient; no built-in finding validation (reporting is basic); dual-use, so isolation is on you.

# start the tool server (ideally inside an isolated container)
python3 hexstrike_server.py           # serves :8888

# wire it into Claude Code as an MCP server
# (the README also documents a JSON config block for Claude Desktop / VS Code)
claude mcp add hexstrike -- \
  python3 /path/hexstrike_mcp.py --server http://localhost:8888
Enter fullscreen mode Exit fullscreen mode

PentAGI: the platform

GitHub logo vxcontrol / pentagi

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

PentAGI

Penetration testing Artificial General Intelligence

Join the Community! Connect with security researchers, AI enthusiasts, and fellow ethical hackers. Get support, share insights, and stay updated with the latest PentAGI developments.

Discord⠀Telegram

vxcontrol%2Fpentagi | Trendshift vxcontrol%2Fpentagi | Trendshift

vxcontrol%2Fpentagi | Trendshift vxcontrol%2Fpentagi | Trendshift vxcontrol%2Fpentagi | Trendshift

Table of Contents

Overview

PentAGI is an open-source, self-hosted multi-agent system for autonomous penetration testing. You describe a target and a goal in plain language; a team of specialized AI agents plans the work, runs 20+ professional tools such as nmap, Metasploit…




An orchestrator analyzes the target and spawns specialists (researcher, developer, executor) plus helpers (searcher, memorist, reporter, reflector…). Tools run in isolated Docker sandboxes. Memory: Postgres + pgvector, optional Neo4j graph via Graphiti. Observability: Grafana/Prometheus, optional Langfuse. LLM: 10+ providers, model choice per agent.
Pros: most product-like (web console, Automation + Assistant HITL, MD/PDF reports), long-term memory, per-agent models.
Cons: heaviest to deploy; fewer tools than HexStrike.

# representative: see the repo for the current compose setup
git clone https://github.com/vxcontrol/pentagi && cd pentagi
cp .env.example .env         # set ANTHROPIC_API_KEY and friends
docker compose up -d         # web UI, Postgres, (optional) Neo4j
Enter fullscreen mode Exit fullscreen mode

The rest of the field

  • Shannon: white-box by design. It starts from the target's source to map attack vectors, then proves them with real exploits ("no exploit, no report"), with the repo mounted read-only in an ephemeral Docker worker. Others touch code too (Strix runs SAST as one tool in a mostly black-box agent, RedAmon clones the repo to triage and fix), but Shannon is the one built source-first rather than bolting static analysis onto a black-box loop. AGPL-3.0 matters if you're commercial; needs source access; and provider-side safeguards can interrupt a run, so clear the Anthropic/OpenAI legitimate-tester verification first.
  • RedAmon: recon → exploit → post-exploit → triage → code-fix → GitHub PR, with a Neo4j attack graph, a docker-broker socket proxy, and human approval gates on high-impact tools. Heavy (~200 to 250 GB for a real server) but closes the offense→defense loop.
  • PentestGPT: USENIX-2024 pedigree; multi-stage pipeline; its agentic mode drives Claude Code or Codex as the execution backend.
  • Pentest Copilot: browser-driven agent, 16 tools incl. Burp, subagent parallelism, and it exposes an MCP server that clients like Claude Code drive. Its MCP token is effectively local admin, so mind the risk.
  • Nebula: a desktop "harness" with a Butler oversight layer, selective memory, Docker/Podman or host mode, optional OpenVPN egress.
  • AI-OPS: built for mid-size/local models. An API container (agent+tools) + CLI, litellm, with a default-deny command allow-list enforced in code. Works with any OpenAI-compatible or self-hosted endpoint via litellm.
  • hackingBuddyGPT: a builder kit. Express a new experiment in a few dozen lines; explicit planner/executor and RAG patterns; a Docker-fleet launcher for regression runs.

Don't build on archived offensive tooling. CAI has 9.8k stars but the repo is archived; unmaintained attack tooling rots as dependencies age. Patterns only. Reaper isn't an agent at all; it's a MITM proxy, good "hands," not a brain.

Pick by what you need

If you need… Reach for
An autonomous agent working in minutes Strix
The widest tool arsenal under your own model HexStrike AI
A platform with UI, memory, human-in-the-loop PentAGI
Source-first white-box, exploit-proven Shannon
Offense → defense with an auto-fix PR RedAmon
An assistant that drives Claude Code / Codex PentestGPT
To run on local / mid-size open models AI-OPS, hackingBuddyGPT

Where Claude fits

The recurring lesson: tool-calling is the bottleneck, and a strong model removes it. That makes every one of these markedly more stable on Claude than on a small local model.

Framework Claude integration Ease Note
Strix STRIX_LLM=anthropic/... ⭐⭐⭐ drop-in
HexStrike MCP in Claude Desktop/Code ⭐⭐ Claude is the whole brain
PentAGI ANTHROPIC_API_KEY / WIF ⭐⭐ per-agent models; WIF needs no static key
Shannon BYO key (harness) ⭐⭐ white-box; provider cyber-verify advised
PentestGPT Claude Code as backend ⭐⭐ drives your Claude Code
Pentest Copilot Claude Code / MCP ⭐⭐ exposes MCP server
AI-OPS litellm ⭐⭐ built for local LLMs

If I were starting today

Shortest path to a working autonomous agent: Strix + a strong model. Keep HexStrike as the arsenal you drive from your own client when you need breadth. Stand up PentAGI when a one-off run becomes a standing service. Pull Shannon in for the white-box view your black-box agent can't see. Everything else is a specialist: know it exists, reach for it when the job fits.

And whatever you run: authorized targets only, in isolation. A sandboxed microVM, egress locked to the target. The capability is real now; so is the responsibility.


Sources: each project's GitHub and docs (linked above), plus the Check Point writeup on HexStrike, and the curated lists awesome-ai-pentest and Offensive-Security-AI-Models. Stars/dates verified via the GitHub API on 2026-10-26.

Top comments (11)

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The scope-enforcement section is the part most surveys skip — star counts tell you nothing, but 'prompt-only' versus 'code pre-call' tells you what happens the moment an agent goes sideways. The open ground you name, an allow-list plus egress lockdown, is the real takeaway: a sandbox contains the blast radius but does not stop a confused agent burning its budget inside the allowed box. In your survey, did any framework do per-call scope checks that track the target itself, like blocking a new host mid-run, rather than just the command?

Collapse
 
mkarytka profile image
Mikalai Karytka •

Great question, and it's the one that matters. Short answer from the survey: no.

The one framework doing an in-code pre-call check (AI-OPS) gates the command, not the target. It asks "is this tool allowed to fire," not "is this host still in scope." Nothing I looked at does a per-call, target-aware check that would catch a new host appearing mid-run and block it.
The closest primitive is RedAmon's socket-proxy broker, which at least brokers connections rather than commands. But that, and the egress locks in Strix/PentAGI/Nebula, still contain at the network layer: they bound what the box can reach, they don't assert "this specific new host is out of scope, refuse" on the call. Containment, not a scope check, which is exactly the line you drew.
This bit me first-hand. The agents follow redirects, including cross-host 302s, without treating the hop as a risk at all. What actually refused an out-of-scope redirect was the egress policy I'd agreed up front, plus the provider-side safety ("not a verified security research engagement"). Neither is the tool asserting scope. The tool was happy to keep going. So target-aware per-call scope is still open ground, same as the in-code allow-list.
On your budget point, that's a second gap worth naming on its own. Egress bounds where it can go, not how much it can burn inside the box. The only brakes I saw on an agent thrashing within scope were coarse step or time caps and a human watching. "You're in bounds but clearly lost, stop" doesn't exist yet either.
Token economy was never a goal of this experiment, so I have no hard numbers there. The aim was preparatory lab work kept as close to a real engagement as I could get, not cost optimization. And honestly, work like this really should run on a controlled local model in a well-controlled environment to begin with, where you can watch the agent, cap it, and reproduce a run without a hosted model's budget and safety behavior sitting in the middle of what you're trying to measure.

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The redirect example is the sharpest evidence in the whole answer — a cross-host 302 is a new target decision the agent never made, yet nothing treats it as one. That also reframes where the check belongs: if the hop itself required re-authorization against scope, the redirect would be caught without a whole new layer. And the 'in bounds but clearly lost' gap pairs with the budget point — a step cap counts motion, not progress, so would a goal-progress check (are we closer to the objective than N steps ago) be the brake that actually fits, or does that just move the judgment call into the checker?

Thread Thread
 
mkarytka profile image
Mikalai Karytka •

Caveat first, I ran these tools, I didn't build them, so this is a user's take more than a design one.
The hop-reauth idea is the one I'd put money on, mainly because it needs no judgment. Is this host in the authorized set, yes or no, deny by default. That's something a tester can actually verify. The progress-check side is more interesting, but it's past what I can tell you from just running things. My gut's the same as yours: it shoves the hard call into the checker, and progress in this work isn't steady enough for "closer than N steps ago" to mean much. That's me guessing, not measuring.
What I'd stand behind from running them is that the dumb signals are the ones I trust. Step and time caps, plus catching when a run stops turning up anything new. Whether a smarter brake is worth building is a question for whoever builds it. If that's you, I'd read whatever you come up with.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The dumb signals earn trust precisely because they're verifiable — a host either is or isn't in the authorized set, and a cap either fired or it didn't. One definition I'd still want pinned down: what counts as 'stops turning up anything new' — no new hosts, no new findings, or no new anything for N steps?

Thread Thread
 
mkarytka profile image
Mikalai Karytka •

"No new findings" is what you care about but the worst brake: findings are sparse and bursty, and only count once validated, so you'd kill runs right before the payoff. "No new hosts" is too blunt, since real progress often happens deep in one host, an auth flow or param fuzzing, with no new host in sight.
The definition I'd use is at the action level: have I seen this (method, normalized URL, param set, response class) before, yes or no. A new endpoint, param, status class or auth state counts. Re-issuing a request you've already made doesn't. Same checkable test as host-in-set, just on the action.
And I'd make it a flag, not a kill: "no new surface in N steps, come look," not "terminate."
One more thing worth adding: the input prompt moves this more than any brake does. Same tools, same target, everything else equal, and the way you phrase the prompt decides how deep the scan goes and whether a given bug even surfaces. That's why none of these questions has a single right answer, there are just too many moving parts to pin one down. Good topic to chew on though. :)

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The flag-not-kill framing is the part I'd keep — a bursty payoff means a kill rule will always fire at the worst moment. And the prompt point reframes the whole brake debate: if phrasing decides depth, then the brake is guarding a run whose shape was set before it started. Do you find a prompt that consistently drives deeper also produces more validated findings, or just more surface to sort through?

Thread Thread
 
mkarytka profile image
Mikalai Karytka •

Honestly, neither. Depth and validation are different axes: the prompt drives depth, but whether it comes back validated or just noisy is about having a PoC step, not the prompt. Deeper mostly changes which class of bug surfaces, not the count.
We've stayed abstract this whole thread, so let me flip it: what are you actually running this on? Which models, what kind of targets? I've seen the depth effect swing hard between a local 7-14B and a frontier model on web targets, and I'm curious whether yours lines up.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Fair flip — and honestly, I don't have a lab answer to give you: this thread has been conceptual on my side, I haven't run these frameworks against live targets, so I can't claim my depth effect lines up with your local-versus-frontier split. Your PoC point still stands though — if validation is a separate step, depth just decides which class of bug you sort through, not whether it survives. Is that PoC step the thing you'd standardize first if you were benchmarking these frameworks, or does it vary too much by target to compare?

Thread Thread
 
mkarytka profile image
Mikalai Karytka •

Yeah, PoC validation first. Without it you're not benchmarking findings, you're benchmarking how much each framework talks. Lock the validation gate and everything downstream — depth, model, prompt — becomes measurable instead of a pile of unverified leads. That's the one control I'd fix before touching anything else.
Good thread, I'll leave it here.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Agreed — lock the validation gate first, and depth, model, and prompt all become measurable instead of unverified leads. Good thread, thanks for walking through it with that honesty about it being a user's take.