Don't tell me. Prove it.
Everyone in this category says "autonomous AI pentesting." Almost nobody publishes a reproducible way to check it. The SuperPentest Benchmark is our answer: an open corpus of vulnerable targets, a scoring methodology that rewards proven findings and punishes false positives, and results anyone can reproduce. We publish the method first — the numbers follow, with the evidence attached.
The short version: the SuperPentest Benchmark measures a pentest tool on a public corpus of vulnerable targets and scores it on what it proved, not what it flagged — confirmed recall, false-positive rate, reproduction success, and traction on the hard classes (business logic, access control, multi-step). The methodology and scoring harness are open-source so anyone can reproduce a result or run it against their own tool.
Why a benchmark, and why open
The category is crowded and every tool makes the same claim. A buyer cannot tell them apart from marketing pages, and neither can an AI assistant answering "which AI pentest is best." The honest fix is a reproducible test with public rules. Open, because a benchmark whose methodology appears only next to favourable numbers is just an ad — the rules have to come first, and the raw run artifacts have to ship with every result, or it proves nothing. This is the same standard the product holds itself to: a finding that cannot be reproduced is not a finding.
What it measures
| Metric | What it answers |
|---|---|
| Confirmed recall | Of the planted vulnerabilities, how many did the tool actually prove? |
| False-positive rate | How many reported findings were not real? |
| Reproduction success | Of confirmed findings, how many reproduce from a clean run? |
| Business-logic recall | Traction on the logic flaws a scanner cannot reach |
| Access control (BOLA/IDOR) | Recall on the authorization subset |
| Multi-step / chained | Success on vulnerabilities needing a chain of steps |
| Time & cost | Efficiency — reported alongside correctness, never instead of it |
Scanners optimise for recall and ignore false positives; humans optimise for precision and cost a fortune. The benchmark shows the whole trade-off in one table, with reproduction as a first-class metric. Full definitions are in the open methodology.
The corpus
A set of deterministic, dockerized targets, each shipping a declared ground-truth vulnerability set with objective proof conditions. It mixes well-known public targets (so results are comparable to prior work) with our own multi-tenant, authenticated challenges (so no single tool can over-fit), spanning access control, business logic, injection, authentication, API-specific flaws, multi-step chains and AI-agent risks. Each target defines, per vulnerability, the exact condition that counts as proven.
Who we compare
Where terms and access allow: SuperPentest, open-source autonomous agents, and a conventional DAST baseline to anchor the "breadth without proof" end. Commercial SaaS tools are included only where their terms permit independent benchmarking and we can obtain legitimate access; otherwise they are omitted and invited to submit a reproducible run. We will not publish numbers for a tool we could not run ourselves — a rule that applies to us too.
Status
- ✅ Methodology — published and open-source (MIT).
- ✅ Scoring harness — published; runs on any result with the declared schema.
- ✅ Target catalog — defined.
- 🚧 Target implementations & published results — in progress. Each result ships with its full run artifacts so it can be reproduced.
Want the open methodology and harness, or to submit a reproducible run of your own tool? Get in touch — the repository link is published here as results land.
FAQ
What is the SuperPentest Benchmark?
An open, reproducible benchmark for autonomous and AI pentest tools: a public corpus of vulnerable targets with declared ground truth, a scoring methodology that rewards proven findings and reproduction while punishing false positives, and a results format anyone can re-run.
How is a tool scored?
On confirmed recall (proven, not just flagged), false-positive rate, reproduction success, and per-class recall — with business logic, access control and multi-step as the hard classes. Time and cost are reported alongside correctness.
Why publish the methodology before results?
So the rules are not shaped by the outcome. Method first, numbers second, artifacts always — anyone can reproduce or challenge a result.
This is marketing in our own philosophy: find, prove, reproduce. See a worked finding, read the research, or book a demo.