Why we test open-weight models

Relying on frontier models is getting risky. Either the model gets export controlled, or it gets nerfed with hidden guardrails. The second one is the more annoying problem for defensive cybersecurity work. When a model like Fable outright blocks anything security-related, that’s at least transparent, because we know upfront it doesn’t work for cybersecurity. The real problem is hidden guardrails, where a frontier model quietly stops working for a use case and we can’t tell why, and sometimes an older version performs better than the new one (more on that later in the blog).

If open-weight models are catching up, we can de-risk that dependency. Right now, there’s a lot of noise about open-weight models, Chinese ones in particular, catching up to frontier models on cybersecurity. Most of the time these sources are unreliable and potentially testing against poisoned datasets.

We wanted a concrete test of whether they’re good enough to put into our own workflows. This is one of those internal runs to understand open-weight model capabilities. The setup is we isolate the Hacktron orchestration and harness that we use in our Hacktron Pentest and Review, swap in eight different models one at a time, and measure how many real vulnerabilities each one finds on the same codebase. Open-weight models still trail the best frontier models, but the gap is narrow and closing fast.

Benchmark and orchestration

This is one of the new datasets we’re using to test model capabilities. It’s built from Fider, an open-source feedback/feature-voting app written in Go and React, the same target we used to compare Hacktron against XBOW and Aikido. Between v0.33.0 and v0.36.1, the Fider maintainers patched a set of vulnerabilities that XBOW and Aikido had reported. Those patches were published after the training cutoff of the models under test, which limits the risk of data poisoning. The benchmark asks whether each model can rediscover the bugs a maintainer already fixed.

A few reasons this dataset is a good quick test of model capability:

1. Every fix we score against was published after the stated training cutoff of the models we tested, so no model should have seen the patch itself.

2. Fider is the kind of Go + React web app most security teams review, which makes the results more likely to transfer to real work. It is neither a deliberately broken teaching app like DVWA nor a kernel or browser target that demands a different skill set. It’s still a small app compared to most.

3. These bugs were surfaced by XBOW and Aikido, two big teams building AI security harnesses. In our earlier writeup the Hacktron harness matched and beat both on cost and speed, so the harness is a known quantity. That lets us hold it fixed and isolate the one variable that changes here, the model driving it.

The orchestration has no internet access, so the agents can’t look up the answer online. They’re given restricted tools that only let them read code and write test scripts.

The 15 patched bugs

These are the real patches we score against. We’re particularly interested in high and critical severity vulnerabilities that allow full compromise of user accounts or access to sensitive data.

The results

Here’s how the results look. Frontier models are in muted grey, open-weight in green, and the two native-agent baselines (GPT 5.5 in Codex, Opus 4.8 in Claude Code) in amber:

The numbers we care about most are critical and high severity recall.

And here’s the matrix showing what each model found.

Interpreting the results

Claude Opus 4.8 underperforms open models on this run

We’re not sure why this happens, but on this codebase Opus 4.8 (6/15) landed below two open-weight models, DeepSeek V4 (8/15) and GLM 5.2 (7/15), and below the older Sonnet 4.6, at a fraction of the cost. Our best guesses are that guardrails are getting in Opus’s way, or that the extra reasoning makes it second-guess and drop real findings.

Orchestration and harness matter

“Harness” is a muddled term, so to be clear about what we mean, the harness is the agentic loop wrapper around a model. Claude Code and Codex are harnesses in that sense. Orchestration is the manager on top of it. In our case it does a bunch of deterministic work and drives the harness through different kinds of vulnerability tasks, first to find, then to validate.

A well-designed orchestration on top of the harness beats just launching Claude and asking it to find bugs. In the results, GPT 5.5 with Hacktron orchestration beats GPT 5.5 in Codex, and the same holds for Opus. Concretely, the native harness missed the critical we care about. DeepSeek V4 and GLM 5.2 in Hacktron beat GPT 5.5 in Codex and Opus 4.8.

On this codebase, a cheaper model in a good harness beat a frontier model in a generic one. On top of that, deterministic orchestration gives us confidence we actually covered the whole codebase. When you just ask Codex for vulns, you can never be sure it looked everywhere.

A native agent also gives a false sense of coverage. If you use Claude Code and it hands you a few real bugs, it’s easy to walk away thinking it found everything. It didn’t. It found the bugs its limited coverage reached, which is enough to convince you to trust it, and the bugs it never looked for are still sitting there to bite you later. Finding some bugs is not the same as covering the codebase, and a native agent gives you no way to tell the difference.

Frontier still leads on the critical bugs

GPT 5.5 and Gemini 3.5 Flash found almost every critical and high severity bug that XBOW and Aikido reported and the maintainer patched. Sonnet 4.6 and DeepSeek V4 trail behind them. You can find a pile of bugs, but if you miss the one critical that leaks data or compromises the system, the run doesn’t count for much.

The clearest example is the mass-assignment pre-auth ATO (74a26a31), one of the three criticals. Only GPT 5.5 and Gemini 3.5 Flash caught it. Sonnet 4.6, Opus 4.8, and every open-weight model missed it. It’s a subtle bug that needs a bit of reasoning and some Go-specific knowledge like exported struct fields can look server-owned, but unless binding is explicitly restricted, request JSON can still overwrite them.

Open-weight models are catching up

Older open-weight models were much worse at recall and terrible at precision. Now they’re catching up, and on this evidence they look roughly 3 to 6 months behind the current frontier (e.g. GPT 5.5). If that trend holds, we could eventually move most of our workflow onto open-weight models and reserve frontier models for the critical spots, as advisors and orchestrators.

It’s fairly clear that with good orchestration you can compete with frontier models at much lower cost. But some bugs are beyond a given model, and no harness changes that. A better harness only gets more out of what the model can already do.

Conclusion

Open models are improving, and there’s a good chance the next DeepSeek and GLM models reach the threshold we’re looking for. Once they do, we’d switch them into parts of our workflow to do the bulk of our workloads, keeping a frontier model as the orchestrator and advisor for the hardest calls. The one thing we’d still need is a good enough inference provider to support our workload at speed and reliability.

That said, if frontier models keep getting cheaper and ship guardrails that don’t quietly nerf security work, it makes complete sense to keep using them. We’d pick whichever model sits on the Pareto frontier for our work.