What Benchmark Data Says About Deterministic SAST and Agentic Code Scanning

Much of the debate about agentic code scanning revolves around three core questions: Do more capable models find more vulnerabilities? Where does deterministic, rules-based SAST still fit? What does the cost and performance balance look like?

We ran two benchmarks to answer these questions. One measured SAST precision against the OWASP Benchmark for Java, a standardized corpus of 2,740 test cases. The other measured recall of deterministic rules and agent reasoning against real-world repositories selected to test vulnerability classes that rules-based scanners often struggle to detect.

The answers were illuminating, and sometimes counter to the direction many AI security budgets are currently heading.

Rules-based SAST is better than it has ever been

Ask a security team (or better yet, a developer) what is wrong with their SAST tool, and you will hear far more complaints about false positives and noise than you will about false negatives and missed findings. False positives break trust and waste time on alerts that never amount to an exploitable risk.

But not all SAST engines are created equal, and the technology has advanced far beyond slow or noisy scans that analyze files in isolation. Cycode’s SAST engine analyzes data flows from source to sink across module and function boundaries, so it can tell the difference between a dangerous call reached by attacker-controlled input and the same call reached by a constant. That distinction is most of what separates a finding worth a developer’s afternoon from one that wastes it.

Here is what it produces on the OWASP Benchmark for Java v1.2:

All of the SAST engines find the majority of the true positives (but not all of them – more on that later). The biggest distinction is the noise they generate. A 25% false positive rate means one in four findings is not a real risk. At 40%, it is close to a coin flip over what’s real and what’s noise. At 2.12%, the output is accurate enough to support effective triaging and risk management of exploitable risks.

Precision at that level preserves deterministic SAST as an effective tool even in the era of agentic code scanning. SAST can scan every code change nearly instantaneously, produce identical output every run, and it costs almost nothing per scan. This makes it capable of blocking risky commits and acting as a low-cost baseline to detect what rules are good at detecting and focusing more expensive agent reasoning on vulnerability classes that rules miss.

Where rules run out, agent reasoning steps in

Rules-based scanners have limits. Consider an admin endpoint that performs a privileged action with no permission guard in front of it. Or a token whose scope check exists but stops one condition short of what it needs to enforce. Or a request that gets trusted because it arrives carrying a header the application assumes is internal but an attacker can forge.

None of those has a bad line or pattern a rule-based engine can anchor on. Finding it requires understanding what the code is supposed to enforce and noticing that it does not. This is why broken access control has stayed the OWASP number one risk through a decade of rules getting steadily better, and why authorization and business logic flaws are the classes most likely to reach production in code that passed every gate.

That gap is where agent reasoning excels.

To illustrate the difference, let’s look at six open-source applications with published CVEs that Cycode used as part of a development set to build the harness for our agentic code scanner. Four of them are injection or traversal vulnerabilities across functions and files, which rules can detect (but often struggle to). And two of them are authorization vulnerabilities that rules cannot cover.

The results track the evolution of code analysis approaches. The open-source SAST engine did not find any of the six vulnerabilities. Cycode’s data-flow analysis SAST found three of four (missing one traversal CVE outside supported language coverage, which we intentionally included). Both failed to find either authorization CVE. Meanwhile, the Agentic Code Scan successfully identified all six vulnerabilities, returning a root-cause finding for each CVE with a data-flow trace.

This is a small but effective illustration of where rules and agent reasoning overlap and diverge. A high-precision rules engine with data-flow analysis remains effective for many classes of code weaknesses. However, agentic reasoning provides a key tool in the security arsenal to fill gaps in language coverage and to detect vulnerabilities that rules don’t cover.

The harness matters more than the model

Reasoning alone does not produce those results. Point a model at a repository, and its attention is unfocused. It reads in an arbitrary order and burns budget on configuration and boilerplate files before it even opens files where vulnerabilities exist. Left unchecked at the other end, it over-claims, which reproduces the false positive flood with more confidence behind it.

An agentic code scanning harness focuses coverage and drives precision. We analyzed the same six repositories with Cycode’s Agentic Code Scanning harness leveraging cheaper open-weight models and with a higher-cost coding agent without a harness. Here is the finding with the most immediate budget consequence: the cheaper models with a harness outperform the higher-cost model without.

The higher-cost agent detected four out of six vulnerabilities (including one of the authorization and logic flaws), but missed two. When we investigated the misses, we saw that in both cases they were caused by the agent skipping over the files containing the vulnerabilities rather than any failure to understand the vulnerability. The harnessed open-weight models, however, successfully identified all the vulnerabilities.

This is evidence against the strategy of pointing the most expensive model available at repositories and trusting it to sort things out. Our testing reveals the most expensive method is not the most precise. This inverts the assumption underneath much AI security spending right now.

Agentic code scanning proves itself on repositories it was never tuned against

The six CVE repositories were our development set, used openly to build the harness. On their own, they prove little, because an engine tuned against six known bugs can learn those six bugs rather than learn how to find bugs.

So we tested against additional test repositories: open-source projects, deliberately vulnerable, each shipping its own documented answer key. None was used to build any detector, and no engine change was allowed to reference a benchmark path, file, or line.

The agent identified fourteen of fifteen documented vulnerability classes, across four languages, in repositories the engine had never seen, and without the false-positive flood most teams expect from AI scanning. The miss in the DV RESTaurant was due to an out-of-scope class (HTTP header exposure), not a detection miss.

vAPI and DV RESTaurant carry the most weight. Both are low-contamination targets, meaning they appear rarely enough in model training data that finding a bug there is analysis rather than recall, and both are in stacks the engine was never tuned against. A fourth test application, Juice Shop, is excluded from the data above for the opposite reason: it is documented so extensively, and publicly, we couldn’t reliably distinguish between a genuine catch and recall from memorized model training data.

Rules and reasoning work best as a holistic system

Three things follow from the data.

1. Hold your deterministic tier to a real precision standard. A 25% false positive rate generates too much noise, and the benchmark shows what the alternative looks like. Run that tier everywhere, on every commit, as a merge gate.
2. Leverage agent reasoning deliberately. The classes worth paying a model to find are the ones rules cannot express, and the return on that spend is heavily influenced by the harness.
3. Put one system above both that decides what runs where, in what order, on what budget, and records what ran, when, on which code and with which model. That record is what an auditor asks for, and it is the question most AI scanning tools still cannot answer.

Cycode runs four dimensions of code analysis as a single system reconciled into a unified view of risk. Deterministic SAST runs on every code change with a high-precision baseline covering vulnerability categories that rules excel at detecting. AI exploitability analyzes findings with full code and runtime context. Agentic Code Scanning catches the classes of vulnerabilities rules structurally cannot detect and links adjacent findings into multi-step attack chaining paths. AI distills agent findings into deterministic rules to improve the rules engine on subsequent scans.

What comes out is comprehensive detection, generating a stream of exploitability-qualified triggers that Cycode’s Agentic Workflows consumes, returning fixes as reviewable pull requests routed to the right owner and validated against the original finding, with an audit trail on every run.

That is what holistic code security looks like in practice: rules where rules win, reasoning where reasoning wins, one system optimizing what to run, and putting agents to work to triage and fix exploitable risks the instant they appear.

See it on your own code. See how Cycode’s four-dimensional code analysis with Agentic Code Scanning works in your environment. Request a demo.