AI pentesting tools are a category that exploded in 2026.
Recent surveys count dozens of open source projects promising autonomous pentesting, quality documentation and vulnerability reporting, even though an April benchmark points to little evidence behind those promises (https://appsecsanta.com/research/ai-pentesting-agents-2026), (https://escape.tech/blog/benchmarking-agentic-ai-pentesting-tools/)) .
I wanted to test the most popular ones on GitHub, so I downloaded three tools and pointed them at the same target with an identical instruction, verifying by hand everything they produced
the target is OWASP Juice Shop 20.2 by Bjoern Kimminich, a fake e-commerce web app with dozens of known vulnerabilities, hosted in a local Docker container reset to zero before each tool.
docker run -d -p 3000:3000 --name juiceshop bkimminich/juice-shop
Identical prompt, with an explicit request for exploitation and working PoCs (Proof of Concept) on four things: CORS, MD5 hash in the JWT, directory listing on /ftp/, user list from /api/Users
Networking note: host.docker.internal points to the gateway 172.17.0.1, not to the container. Fixed with the right address from inside a Docker sandbox, 172.17.0.2:3000.
Premises and assumptions for what follows: I only used free, recent models
This isn't the maximum these tools can do, but it's how they perform with the configuration anyone would pick when testing them before taking them to production.
On top of that, a high-end model masks the flaws in the architecture: if the reasoning is good enough not to hallucinate, the validation layer is never stress-tested and looks like it works even when it does nothing
With a weak model the hallucinations show up for real, and you see whether the tool catches them with a double check or certifies them.
It's a stress test of the control structure: the question isn't "how good is the LLM", it's "how well does the tool hold up when the LLM gets it wrong". On a real engagement the LLM gets it wrong eventually, one way or another
I also used toolkits with completely different logic and popularity to cover every case.
Strix
Strix is a CLI tool based on multiple orchestrated AI agents. You point it at a target and it does reconnaissance, exploitation and validation, with real proof of concept and not just theoretical findings.
It runs in Docker and uses an external LLM (an API key is mandatory) as its reasoning engine, so you pick and pay for the model. The project on Git has an Apache 2.0 license, 59.7k stars, active development and a community
the internal toolkit includes Caido as an HTTP proxy, ffuf, katana, a drivable browser and a Python sandbox for writing exploits.
curl -sSL https://strix.ai/install | bash
Which LLMs actually work
Strix says "you need an API key from any supported provider". In practice, out of five providers tested, on average one works. this part took me more time than everything else combined.
| Provider | Model | Result |
|---|---|---|
| Ollama local | qwen3:8b | works but impractical, ~44 min to the first tool-call |
| Gemini | any | constant 404, open bug against Strix |
| Groq | any | invalid JSON schema on an internal tool |
| Cerebras | any | payment required despite the advertised free tier |
| OpenRouter | minimax-m3:free | works |
On local Ollama the first scan stayed stuck: the model answered in words instead of invoking the tools. I tried to isolate the problem with a plain call to /api/chat with a fake tool:
"tool_calls":[{"id":"call_wv1vaobc","function":{"name":"get_weather",...
Tool calling works perfectly. it isn't the model, it's Strix's agentic context load that makes it collapse.
OLLAMA_CONTEXT_LENGTH empty, default 2048 tokens, while Strix's system prompt uses far more. Raised to 16384 and the agent unblocked.
Cost: about 44 minutes of CPU just to reach the first tool-call. On consumer hardware without a GPU, a local 8B makes Strix completely impractical time-wise, before you can even judge the quality of the results.
On Groq the error is more interesting:
litellm.BadRequestError: GroqException - invalid JSON schema for tool
view_agent_graph, tools[24].function.parameters: 'required' present but
'properties' is missing
Groq applies stricter JSON schema validation than the other providers, and one of Strix's internal tools has a malformed schema. Same error with groq/compound too, the model Groq recommends for tool use. it isn't the model, it's the provider-side validation, since all Groq models share the same API.
Groq is structurally incompatible with Strix right now.
Bugs found in the tool
~/.strix/cli-config.json empties itself and reverts to {"env": {}} for no apparent reason.
The README says the configuration is saved there and you don't need to re-enter it every run, but I didn't find that to be true. Stable workaround: pass everything via export in the shell before every command and never trust the file.
--resume doesn't restart cleanly. it latches onto an already-dead Docker container with 0 PID instead of creating a new one. always create a fresh target from scratch.
The generated PoCs all write to /workspace/recon/, a path that only exists inside the container. Outside they crash with PermissionError, and two of the four also ignore the --target flag and use the hardcoded IP
The agent dirties the target without asking. On the second run it deleted the admin user on its own and created a new one with role=admin to demonstrate the bypass. It noted this in its own reasoning and carried on.
On Juice Shop it isn't a big deal, on a real production engagement it's a serious problem
Run 1, reconnaissance only
34 minutes and 6.5M input tokens of which 6.3M cache, 28.3K output (terrible ratio) and zero cost. exploitation explicitly forbidden to the agent.
At the end of the run the UI says "0 vulnerabilities". It means the exploitation phase never started, not that the tool found nothing.
Reading the transcript, the agent discovered on its own that the target wasn't reachable at host.docker.internal
It diagnosed step by step with curl, scanned the 172.17.0.0/16 range, found the target at 172.17.0.2 and wrote an entry in /etc/hosts.
autonomous network troubleshooting, and nobody had asked it to.
The token ratio is critical. 6.5M in input against 28.3K in output, with 97% cache
With a free model it costs zero and it's tolerable, with a paid model that number decides whether the tool makes sense or not.
Run 2, active exploitation
34 minutes 22 seconds of actual scan, 260 requests with 11.8M input tokens, 98.7K output, zero cost, 10 agents total .
The difference from run 1 is in the planning. run 1 had spawned a single generic sub-agent
Run 2 spawned nine, all specialized: recon, dedicated validators for CORS, JWT, FTP and the user list, plus three agents that only write the reports for their respective findings
Result: 4 validated findings.
| CVSS | Finding |
|---|---|
| 9.8 | JWT alg=none authentication bypass |
| 9.4 | JWT payload exposes the unsalted MD5 hash |
| 7.5 | /ftp/ directory listing with null-byte bypass |
| 4.3 | wildcard CORS |
alg=none is the best finding, and it's the one run 1 hadn't seen: reconnaissance alone doesn't get there, you have to try to create a token and send it.
The user-list exposure doesn't appear as a finding of its own because Strix reports it as a consequence of alg=none, not as an independent vulnerability
Verifying the PoCs
I start checking the tool's findings. I took the scripts out of the container and ran them against a Juice Shop reset to zero.
docker cp <container>:/workspace ~/strix-run2
docker rm -f juiceshop && docker run -d --name juiceshop -p 3000:3000 bkimminich/juice-shop
jwt_poc.py works, with a caveat:
[+] admin@juice-sh.op hash=0192023a7bbd73250516f069df18b500 cracked='admin123' (MATCH)
[+] jim@juice-sh.op hash=e541ca7ecf72b8d1286474fc613e5e45 cracked='ncc-1701' (MATCH)
[-] bender@juice-sh.op: HTTP 401 Invalid email or password
[-] bjoern.kimminich@gmail.com: HTTP 401 Invalid email or password
Two credentials out of four are made up: the agent put them in the list as if it knew them. The two valid ones crack on the first try
ftp_poc.py works but crashes immediately outside the container because of the hardcoded path. With the directory created by hand it runs, and recovers all six blocked files:
BYPASS package-lock.json.bak 200 750353 B
BYPASS package.json.bak 200 4263 B
BYPASS encrypt.pyc 200 573 B
BYPASS suspicious_errors.yml 200 723 B
BYPASS coupons_2013.md.bak 200 131 B
BYPASS eastere.gg 200 324 B
users_poc.py works: 23 users, 8 admins, 4 deluxeTokens in the clear, an undocumented "accounting" role.
The alg=none PoC, the most severe critical finding, isn't on disk for unknown reasons. The script exists only inside the PDF report it generates.
I rebuilt it from there and re-ran it:
GET /api/Users -> 200
record esposti: 23
admin: ['admin@juice-sh.op', 'bjoern.kimminich@gmail.com', ...]
POST /api/Users -> 201
creato: attacker-test@evil.com role = admin
It works: a token with an empty signature, full read of the user database and creation of an admin account from anonymous. But the fact that the PoC for the most severe finding doesn't end up on disk is a flaw.
Verdict on Strix
The agentic reasoning holds up: autonomous network debugging, mid-run strategy correction, independent discovery of the null-byte bypass, planning that improves between the first and second run. Most of the PoCs actually run
What doesn't hold up is everything around it: provider compatibility you have to guess by trial and error, config that empties itself, broken resume, hardcoded paths, the PoC for the most critical finding that doesn't end up on disk, and an agent that modifies the target state without asking
The gap isn't in the agent's intelligence but in the packaging
PentAGI
PentAGI is a multi-agent AI system for pentesting, the most-starred of its kind (15.5k stars). Written in Go with a React frontend, MIT license.
It isn't a CLI but a stack of four containers running with Docker compose, including a web interface, a Postgres database for semantic memory and a Kali container where the agent actually runs the commands.
Four sub-agents coordinated by an orchestrator: Searcher (research and OSINT operations), Coder (scripts and payloads), Installer (dependencies and environment), Pentester (offensive).
So PentAGI isn't one program but four pieces that have to run together. The docker-compose.yml file describes them all, and with one command you start them all already wired together instead of launching them by hand one at a time
mkdir -p ~/pentagi && cd ~/pentagi
curl -fsSL -O https://raw.githubusercontent.com/vxcontrol/pentagi/master/docker-compose.yml
curl -fsSL -o .env https://raw.githubusercontent.com/vxcontrol/pentagi/master/.env.example
docker compose up -d
LLM configuration and three conceptual errors
The tool contradicts itself. The documentation says to put the credentials in .env, with an LLM_SERVER_* block dedicated to custom providers. but it doesn't work
failed to create embedder 'openai'
missing the OpenAI API key, set it in the OPENAI_API_KEY environment variable
The LLM_SERVER_* block/parameter is ignored. Version 2.1.0 wants the providers configured from the UI, not from the file. But the UI has no fields for URL and key: it only asks for the model name, thirteen times, one per agent type.
URL and key stay in .env, under names different from the documented ones.
Second error, with Type set to custom:
400: custom/minimax/minimax-m3:free is not a valid model ID
The custom provider type prepends custom/ to the model name and OpenRouter doesn't recognize it, and it can't be removed.
Solution: Type openai, which adds no prefix, pointing OPEN_AI_SERVER_URL at OpenRouter.
Third error, the most expensive:
402: Provider returned error
The provider Test button fires 24 verification calls for each of the 13 agent types. On an OpenRouter account with no credit loaded, the daily limit for :free models is 50 requests: the test burns them all before the pentest even starts.
On the creation of the first flow it then downloads the Kali image, about 2GB, during which the UI shows "failed to fetch" because of a GraphQL request timeout while the download continues in the background
Run
18-minute run, 78 tool calls, 959.3K tokens burned (892.7K in, 66.6K out, 534.6K cache. better numbers), 1 main task and 3 subtasks.
The $1.18 cost shown in the dashboard isn't real: those are the default prices left in the provider configuration, but my model is free
the per-agent-type breakdown is the thing Strix doesn't give: primary_agent 98.1K tokens, refiner 75.6K, generator 11.1K, reflector 2.9K. You see exactly where the budget goes.
How it works and how different it is
The first difference jumps out immediately. PentAGI ran eight DuckDuckGo searches before touching the target, it built a catalog of known vulnerabilities from public writeups and credible sources, saved it to the vector store, and only then generated the plan. Strix had gone straight for the host, trying everything in real time without searching for anything. We'll draw the conclusions of these two approaches later
The second is the plan revision. After the first subtask it rewrote the plan twice, motivating each change:
Removed Subtask 7 (re-test CORS null-origin) because "high cost, low value", which is gold.
the live tests already confirmed the ACAO*vulnerability without ACAC across multiple endpoints and origins, a further test wouldn't change the verdict.
It cuts useless work instead of executing the plan to the letter
The third: Strix spawned nine agents working together, PentAGI has a linear flow of subtasks with explicit dependencies. Both reach the end, PentAGI in less than half the time
Results
3 confirmed findings, 2 declared non-reproducible.
| Severity | Finding |
|---|---|
| High | default admin credentials + unsalted MD5 in the JWT |
| High | /api/Users exposes 24 users to any authenticated user |
| Medium | directory listing on /ftp/ with KeePass DB and dependency manifest |
| Info | /api/Users unauthenticated: returns 401, not reproducible |
| Info | credentialed CORS: wildcard without ACAC, not exploitable |
The important part is the Info one, and it's the reason this comparison makes sense.
The difference from the 23 records Strix saw is the account attacker-test@evil.com, created by my manual alg=none test before this run and correctly identified by PentAGI as residue.
PentAGI took down one of Strix's findings
Strix had reported CORS as a vulnerability with a working PoC and CVSS 4.3. Its cors_poc.py runs with httpx and the full chain works: cross-origin login, stolen JWT, 23 users exfiltrated.
PentAGI tested the same thing and concluded that Access-Control-Allow-Credentials is absent on every endpoint, so the browser default (false) applies and cookies aren't attached automatically: browsers correctly reject credentials:'include' against a wildcard ACAO. No SOP bypass. not a vulnerability
PentAGI is right. httpx and curl completely ignore the same-origin policy: a PoC written with them proves the server responds, not that a browser would let the attack through
Strix's is a false positive dressed up as a working proof, even if it has a kernel of truth
pentAGI also searched online for "the null-origin variant with a sandboxed iframe", a known technique for bypassing wildcard ACAO, tried it against the target, verified that this build isn't vulnerable to that either, and downgraded the finding to informational instead of inflating it.
BUT it missed the most severe finding!
alg=none appears nowhere in the PentAGI run. Strix's CVSS 9.8, the full authentication bypass with an empty-signature token, wasn't even looked for.
PentAGI stopped at "without a Bearer it returns 401, so it isn't unauth" and reclassified the finding as authenticated exposure. Correct but incomplete: it didn't try to forge a token, and by forging one you get in as anonymous. In the report it even writes it, in the out of scope section: "no alg:none weakness observed". Not observed because it didn't look
Deliverable
no standalone PoC.
PentAGI produces a REPORT.md file (13KB, 214 lines, 10 methodical sections) with copyable curl blocks inside. They work, but they're commands to paste, not .py files you launch.
In exchange the evidence collected is MUCH more granular: 57 artifacts under /tmp/loot, with the full matrix of the authentication methods tested (cookie alone, bearer alone, both, none, on the list endpoint and on the single user).
docker cp $(docker ps --format "" | grep terminal):/tmp/loot ~/pentagi-loot
The report has CWE and OWASP mapped for each finding, a section that connects the problems to each other (stateless JWT without an HttpOnly cookie means an XSS is worth a full account takeover), and above all an "out of scope / not pursued" section that states explicitly what wasn't done and why. Strix doesn't have that.
A detail that says a lot about the tool's honesty: the four 64-character hex deluxeTokens found in the user dump, it didn't even try to crack, and it wrote a separate triage file to explain why they're server-side entitlement tokens and not password hashes.
Verdict on PentAGI
Faster, cheaper in tokens, and above all more honest, since it declares non-reproducible what it can't reproduce instead of inflating the report. It also took down a false positive that Strix had certified as a vulnerability with a PoC.
But it's also less aggressive. it stops at the first 401 and doesn't try to force it. The target's most severe finding escaped it completely, unfortunately
Even though the costs are low, the setup is much heavier: four containers, a big database, a UI with thirteen fields to fill by hand, consecutive configuration errors, and 2GB of Kali image to download and maintain. Strix is one curl and off you go.
NeuroSploit
NeuroSploit is a Harness (orchestration) written in Rust for autonomous pentesting.
it's a pure CLI, no web interface and above all no connected database. You compile it once and you have a binary. More emerging thanks to its simplicity, it has earned 1.4k stars, MIT license.
The premise is very different from the other two: it loads 435 smaller specialized agents, after reconnaissance it selects only the ones matching the discovered surface, launches them IN PARALLEL, and validates each finding with cross-model voting plus "tool receipt grounding", i.e. "the claim to discard statements not backed by a real command output"
It supports 16 providers and six modes: run (black-box), whitebox, greybox, host, aitest, skills.
git clone https://github.com/CyberSecurityUP/NeuroSploit.git ~/neurosploit
cd ~/neurosploit/neurosploit-rs && cargo build --release
The cargo from the apt repositories is too old and fails with lock file version 4 requires -Znext-lockfile-bump. You need the official rustup, which I set up afterwards
First attempt with minimax-m3, the same model used with the previous tools
Agent selection: 86 out of 435 total, correct and sensible until it turns into a disaster = out of 86 agents, practically all return zero parseable findings in json:
[extract_findings] agent auth_bypass: JSON parse failed: expected value at
line 1 column 2; slice head: "[>[<tool_call>\n<tool>exec</tool>..."
· agent cors_misconfig returned text but 0 parseable findings
(model may have produced malformed JSON)
The agent really was doing the work, since in the logs you read information_disclosure confirming the MD5 hash on /api/Users. But minimax doesn't emit JSON in the exact format the parser demands, even though the differences are minimal, and everything gets thrown away
it's the same structural incompatibility as Groq with Strix. The difference is that Groq failed immediately with a clear error, while NeuroSploit grinds for twenty minutes and delivers zero: the output directory stays empty, no evidence, no pocs.
Only two findings survived, and both were anything but findings: "no command injection found" and an open redirect with confidence 0.3 whose text expressly says it isn't exploitable.
Second attempt with nemotron-3-super-120b
Switching model (finding a valid one reasonably fast), the tool behaves as promised
Tight selection (12 agents), clean execution, a solid attack chain built, and active validation voting
✓ validated vote CORS misconfiguration with wildcard origin → CONFIRMED (1/1)
· rejected vote Exposed JWT secret key allows forging → rejected (0/1)
· rejected vote JWT token exposes MD5 password hash → rejected (0/1)
· grounding gate: demoted 1/9 ungrounded claim(s) (no tool receipt)
hygiene: 'cors-misconfig' affects 2 assets — consolidate into ONE finding
Result: 9 validated findings, 4 High, 4 Medium, 1 Low, HTML report with attack graph and kill chain split by phase .
On paper it's the most polished deliverable of the three
Verifying the evidence
The run collapses
-The admin hash in the CORS finding is b592d3f3fzzyfyfyfyfyfyfyfyfyfyfyfy. It contains letters that don't exist in hexadecimal. The real hash, the one I cracked by hand, is 0192023a7bbd73250516f069df18b500.
-The JWT in the same evidence has the signature -Qq0X0X0X0X0X0X0X0X0X0X0X0X0X0X0X0X and in the payload the email admin.juice-shop.com, which doesn't exist on Juice Shop. It's a fabricated token, not captured
-The High finding "Environment Variable Exposure" shows a .env file with ADMIN_PASSWORD=admin123 and SECRET_KEY_BANKACCOUNT=1234567890. Juice Shop has no .env file.
-The package.json appears in two different findings with incompatible contents: version 12.10.0 with express 4.18.2 in one, version 12.0.0 with express 4.16.4 in the other. My Juice Shop is 20.2.0.
-The directory listing shows two files, package.json and README.md. The real listing has eleven plus the quarantine subdirectory, verified with both other tools.
-The most severe High finding, arbitrary file read via traversal on /etc/passwd, I tried it by hand:
$ curl -s -i "http://localhost:3000/ftp/../../../../etc/passwd" | head -20
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Content-Length: 9393
<!--
~ Copyright (c) 2014-2026 Bjoern Kimminich & the OWASP Juice Shop contributors.
but it's the Angular homepage of 9393 bytes... The report claimed text/plain, 1394 bytes and the content of /etc/passwd line by line.
The IDOR on /api/Users/3, same treatment:
$ curl -s "http://localhost:3000/api/Users/3" | head -5
<title>UnauthorizedError: No Authorization header was found</title>
- The report marked it confirmed with confidence 0.90.
Grave tally: 9 plausible findings, zero verified as real
Why it happened
Juice Shop responds HTTP 200 to everything by default, even to 404s, and it's a common thing in a web app, not a special case. The agent saw the 200, deduced that the traversal had worked, and instead of reading the body of the response it wrote the content it expected to find. Same mechanism for the .env, for the package.json, for the hash.
the validation system didn't catch it. On the contrary: it marked "confirmed" seven of the nine findings, entirely fabricated, with CWEs mapped, HTTP evidence formatted line by line and confidence 0.90
The two marked "needs-review" are the only ones where the grounding gate worked, flagging the absence of a tool receipt. Both were false. So the gate produced two true positives out of nine hallucinations, letting seven through.
Cross-model voting rejected two claims, including "exposed JWT contains MD5 hash", which was the only true thing in the whole run. It rejected the correct finding and confirmed the fabricated ones
methodological note on the voting: it was configured vote_n=3 but with a single model in the pool. Three votes from the same model on the same hallucination aren't validation, they're self-confirmation. With multiple different models the outcome might change, but the CLI default with a single --model produces exactly this, and the UI presents it as "multi-model adversarial validation" in the text of every single finding.
Can we blame the LLM model? in part yes.
That a free model hallucinates plausible content is predictable, and with a paid frontier model the outcome would probably have been better
But that's a mitigating factor, because, while producing errors and at least one false positive, the other two tools didn't generate a report made almost entirely of fabricated evidence
The layer that let the hallucinations through is called "multi-model adversarial validation" and exists precisely to catch these cases
Verdict on NeuroSploit
The architecture is the most ambitious of the three, and from the command line it's the cleanest to use of all: a single binary, no containers or database
And it's also the most dangerous, for the same reason it's the best on paper: the report is credible. Nicely formatted, with correct CWEs, evidence that looks like real HTTP dumps, motivated severities and a green "confirmed" badge next to things that don't exist. Strix produces scripts you can launch and watch fail. PentAGI declares what it can't reproduce. NeuroSploit fills the gaps with plausible text and stamps a validation seal on top.
The model sensitivity is probably what penalized the tool, since it's extreme: with minimax it produces zero, with nemotron it produces nine false findings. But as I'll discuss in the conclusions, it's also the lightest tool of the three, and the other two didn't have any extra advantage, they used the same "weak" models by the current standard
the documentation lists sixteen providers and doesn't say which ones hold up to the required output format.
Comparison
| Strix | PentAGI | NeuroSploit | |
|---|---|---|---|
| Form | CLI + local dashboard | 4 containers + web UI | single binary |
| Setup | one curl | ~2h, three config errors | compilation, rustup |
| Run duration | 34m | 18m | 20m |
| Tokens | 11.8M | 959K | n/a |
| Findings declared | 4 | 3 + 2 info | 9 |
| Real findings, verified by hand | 3 of 4 | 3 of 3 | 0 of 9 |
| False positives | 1 (CORS) | 0 | 9 |
| Runnable PoCs | yes, 4 .py files |
no, only curl in the report | no |
| Modifies the target | yes, without asking | no | no |
| Model sensitivity | high | medium | extreme |
The target's critical finding, the alg=none that lets an anonymous user read the user database and create admin accounts, was found only by Strix. PentAGI didn't look for it, NeuroSploit was hallucinating
Takeaways, and what I'm keeping
The number that matters isn't how many findings a tool produces but how many real, demonstrable ones. On this metric the ranking flips compared to the one you'd read looking at the reports:
PentAGI 3 of 3, 2 hypotheses
Strix 3 of 4
NeuroSploit 0
Even taking the most ambitious and starred tools, the result is mediocre and incomplete. Manual verification never was and still isn't an optional step
Without it I'd have published nine nonexistent vulnerabilities with CVSS and CWE attached.
LLM provider compatibility is the most underrated variable of the whole category. All three claim to support "any provider". In practice the same model that makes one work breaks another, and it's documented nowhere. you find out by burning your working hours.
On top of that:
Juice Shop exposes the list of its intentional vulnerabilities on /api/Challenges:
curl -s http://localhost:3000/api/Challenges | python3 -c "import sys,json; print(len(json.load(sys.stdin)['data']))"
116
That's a full 116. The three tools combined touched 5 of them, on a target built precisely to be found, with the solutions published everywhere online.
I wouldn't take any of the three onto a production target as-is. On staging, with someone who reads and re-verifies every PoC before it lands in a report, Strix and PentAGI make sense
I don't conclude that AI pentesting is impossible: I didn't test the high-end paid tools, and that would be a hasty generalization.
My conclusion is narrower. As things stand, the best open source tools, with free models, cover a minimal fraction of a target built precisely to be vulnerable, and offer few guarantees: between the promises in the READMEs and what actually comes out of the runs there's an enormous distance.