If a large model finds a 0-day with 90% probability, a small model finds it with 50% probability, and the small model costs 10x less, the small model is the better choice.

In our last blog post we let Claude Opus write a Chrome exploit. Anthropic also partnered with Mozilla to let their Mythos model discover 271 vulnerabilities¹.

But these results were achieved with a skilled operator in the loop. A researcher or skilled engineer who knows what to look for, how to guide the agent’s exploration, how to set up a good harness and goal, and when to interrupt and course-correct. In that setting, large frontier models amplify an expert’s work, and they may be necessary for complex codebases like browsers and kernels.

But 99% of applications do not have the complexity of a JIT compiler. And Hacktron has no human operator guiding it or telling it where to look. How much do large models matter in that kind of workflow?

Recently we reported two 0-days in oauth2-proxy², which we used to benchmark our Hacktron scanning pipeline and compare different models. In that benchmark, smaller models run repeatedly outperformed larger frontier models on cost-to-recall, and we expect the same for most applications.

Security research in 2026

To test whether Hacktron can find real 0-days, we need 0-days discovered independently of our workflow. So Rahul (“iamnoooob”) picked a target and prompted Claude Code, with an additional skill³:

with follow-up guidance:

A prompt like this finding real 0-days is no longer a surprise, but it still amazes me every time I see it.

A skilled human researcher could have done all of this before LLMs. We (humans) didn’t, because of the cost and the scarcity of top-tier talent.

Running an agent is much cheaper than staffing a security team, which is why the economics look so good today. But I believe this is just temporary arbitrage. The research session above cost around $200 in tokens, plus one to two days of work by a professional researcher.

Human-out-of-the-loop

Without a skilled operator, you could loop a large model indefinitely or spin up an agent per file, but costs explode quickly. Even on a relatively small repository like oauth2-proxy, you are already looking at thousands of dollars. Scale that to something larger like Gumroad, and you are easily in the tens of thousands. I can only speculate, but I estimate Mozilla used at least $100,000 worth of tokens to find the 271 vulnerabilities. I wanted to say $300,000+, but to avoid embarrassing myself with an outrageous guess, I picked $100k.

This compounds further in continuous workflows. If you run scans on every PR to catch issues early, then something like Claude Code Reviews⁴ is “billed on token usage and generally averages $15-$25, scaling with PR size and complexity.” At any reasonable scale, you blow past the $200 mark almost immediately.

LLMs are cheap compared to traditional security research, but what matters is where the market, which always optimizes for cost, will settle. What counts as cheap today quickly becomes tomorrow’s baseline. Most of the market will not pay for the absolute best at any price, and it does not solely consist of software as complex as browsers. If we can find the critical bugs in 99% of web apps, that is “good enough,” and the market will converge on cost-efficient setups for that.

Benchmark

We benchmarked our Hacktron pipeline in different model configurations against oauth2-proxy v7.15.0⁵, a widely deployed authentication reverse proxy with over 14,000 GitHub stars. Two zero-day authentication bypass vulnerabilities that we discovered served as ground truth.

Finding A is the health check User-Agent matching bypass in auth_request mode (CVE-2026-34457)⁶. Finding B is the authentication bypass through X-Forwarded-Uri header spoofing (CVE-2026-40575)⁷.

Hacktron workflow

The Hacktron pipeline has six steps:

1. Code parsing builds a structured call graph of the target codebase, identifying entry points and data flows.

2. Context gathering collects and indexes documentation, configuration files, and other relevant sources.

3. Code enrichment annotates code paths with data flow information, authentication boundaries, and control flow characteristics.

4. Structured presentation assembles the enriched context into targeted prompts and presents them to LLMs for vulnerability analysis.

5. Post-processing deduplicates, validates, and scores the findings.

6. Human triage is a minimal targeted review step. We want humans out of the loop, but while we tune false positive detection, we keep it so clients only get findings that hold up.

For this benchmark, we first ran steps 1 through 3 of our workflow and used that as the initial state. Then we ran step 4 plus deduplication with different models. This means we are evaluating the recall performance of the models, since we haven’t run the validation step yet. At this point, we simply care about finding as many true positives as possible.

In step 5 (post-processing, during validation) we would care about precision, since we want to avoid false positives.

Benchmark results

The results are hard to interpret without knowing how Hacktron works, so a few values need context:

• In this test GPT-5.4 costs about as much as GPT-5.4-mini because it produced 10x fewer findings, which reduced how many LLM calls were made in total. I don’t know why GPT is not producing more findings.

• Claude Sonnet 4.6 is much more expensive than Claude Opus 4.6 for the same reason. Sonnet produced a huge amount of potential findings, leading to many more LLM calls overall. This is not inherently “better” because all of them still have to be validated later, incurring even more cost.

• Claude Haiku 4.5 performs similarly to GPT-5.4 nano. The findings seem to be at the upper bound of their capabilities.

• Gemini 3.1 Flash Lite has exceptional results. Here are two hypotheses why that is:

  • Our prompts may be unintentionally optimized for Flash because during development and testing we avoid expensive models and use smaller models like Flash more often.

  • Gemini 3.1 Flash Lite may simply be a really good model⁹.

Dealing with LLM non-determinism

Let’s ignore Gemini 3.1 Flash Lite as an outlier and look at the other results. Both large models (Claude Opus 4.6 and Gemini 3.1 Pro) surface Finding A and B more reliably than the small models, but they still do not consistently surface both findings.

Once you look at the cost, you realize that for the price of a single Gemini 3.1 Pro scan, you can run Flash 14 times. And for the cost of a single Claude Opus 4.6 scan, you could have scanned the code with GPT 5.4 Nano 28(!) times.

GPT-5.4 Nano has only a 10% chance to surface Finding A, but if you run it 20 times, you almost get to 90%. If you have GPT-5.4 Mini with a 70% chance, you can get to 90% with 2 runs:

For a long time we have been running large models at Hacktron, but whenever we evaluate against a real 0-day, we do not really see large models performing that much better. They are also non-deterministic and rarely have a 100% success rate, which makes small models (called multiple times) the better choice.

If you compare the relative cost required to reach 90% for Finding A and Finding B, the small models are “better”:

Context matters

Context matters a lot for Finding A. To see that it is a vulnerability, you need to know the role of OAuth2 Proxy and its two deployment modes, standalone reverse proxy and middleware component¹⁰.

The vulnerability requires reasoning about a second system (e.g. upstream Nginx), interpreting a 200 OK health check response as an authentication success signal. Without that deployment context in the prompt, the model correctly concludes the health check is a harmless dead end, because in standalone mode it genuinely is.

Comparing the raw prompts from cases where the models did and did not find it, the successful prompt includes information from our context engine about the deployment modes. The failing prompt does not mention the middleware role or the delegated authentication at all. In this case, the missing context mattered more than model size.

Conclusion

Large models are powerful, but once cost matters, running a smaller model several times can beat running a large one once. For security audits, we want recall to surface as many potential vulnerabilities as possible, and precision when we validate them.

In human-in-the-loop settings, with our limited human cognitive capacity, it makes sense to use a large model to get the best output in a single run. In our experience, though not measured in this benchmark, large models are more accurate when validating complex vulnerabilities.

But small models can surface the same 0-day vulnerabilities, just less reliably. That reliability can be improved by running the model multiple times for the same or lower cost than a single large-model run.

That is why Mythos finding vulnerabilities by itself does not worry us. We know models get better and that very simple agent loops can find real vulnerabilities. But if you are not getting subsidized by a frontier lab¹¹, cost matters eventually and you need optimized workflows that get the most out of the models. Most applications are not on the complexity level of browsers and kernels where Mythos-level upper bounds are required.

Finding the two vulnerabilities with an operator cost over $200 with Claude Code and 1-2 days of human work. With a basic agent loop over the files, it would have cost thousands. With our optimized Hacktron workflow, we could find them for ~$12. At that price, we can also run many times and get more coverage than running the same workflow with a large model once.

We re-evaluate and tune the workflow whenever new models are released.

If cost matters to you and you are not in the 1% of the market that needs Mythos-level upper bounds, our workflow gets you the same 0-days for less.

If cost is not a constraint, we can run our workflow with any large model you want. Contact sales.