Harness & Hunt: A Hybrid, Role-Driven Agent Architecture for Autonomous Vulnerability Research

[ Abstract ]

A three-month study adjudicating 630 agent-generated vulnerability findings against the code at the reported line. In their data, precision depended more on method than model, the more expensive frontier model did worse on static analysis, and verification is still the bottleneck.

LLM agents can already carry some vulnerability investigations from discovery to exploitation. In one of our tests, an agent decompiled a stripped binary, identified an overflow, and produced a working exploit against a live service. That result led us to a broader question: how often do agent-generated findings survive code-level verification?

Verification requires tracing attacker-controlled data to a sink, checking whether a real request can reach it, and deciding whether the weakness is exploitable. This process determines whether a client can act on the report or will lose trust in it. Today, a human still has to read the relevant code line by line. That manual review becomes the bottleneck in an automated pipeline, yet published results rarely measure it.

This session reports a three-month study of two models across multiple targets, including source code, real Java repositories, stripped ELFs, live web applications, Android APKs, and Solidity contracts. Across these targets, we collected 630 findings and adjudicated each one against the code at the reported line rather than the model's summary.

Precision depended more on method than model. Confirmation-based approaches reached 79-100%, while static analysis of real codebases reached 65-77%. On the static-analysis target, the more expensive frontier model performed worse: 285 findings with 99 false positives, compared with 169 findings and 39 false positives from the cheaper model. These results led us back to verification. A binary true-positive/false-positive verdict proved too crude, and an LLM judge drifted toward "uncertain" as the workload grew. We also document reproducible pipeline failures that looked like clean results.