Freedom invites chaos

Security is seldom the first priority of a new technology, and in the wild west of the Internet, hackers will always rise up to the intellectual challenge of breaking into the latest and greatest. Science fiction’s habit of assuming that a powerful, disruptive technology leads to chaos and destruction is overly pessimistic, but it has a point.

When Robert Morris created the infamous Morris worm in 1988, he simply wanted to see if it could be done. This is perhaps not too different from when an innocent software engineer who hasn’t done any serious hacking before decided to poke into production apps hosted by Lovable.

Code generation tools like Lovable let a new generation of non-technical users create software, and they do it by giving the AI absolute freedom over the code it writes. But just like how the wild west of the Internet had tons of competing browsers (none of them secure) before browser standards became a thing, we are headed into a new wild west of AI-generated code where the rate of code generation far outstrips the human ability to review and secure it.

The freedom to create has always invited chaos in software. Much of the recent work in the security space has tried to rein in that chaos by making it intuitive to only create a strict subset of all possible software and hoping that this subset is secure. “Secure by default” frameworks are a good example. The difference this time is that the number of insecure systems, which used to scale linearly with human capital, will soon scale exponentially with the number of GPU cores.

Scale is the enemy

Great, the amount of code in the world is going to explode. Surely the same AI that generated the code can also be used to secure it, right?

Let’s take a step back and look at what a vulnerability is. A vulnerability can only exist if there is a difference between the intended behavior of a system and the actual behavior of that system. So to prove a vulnerability, you are showing that

and presumably that mismatch gives the attacker some kind of advantage over the system.

So proving that a system is secure is way harder than proving that it is insecure. This doesn’t even take into account the fact that you can’t expect an AI agent to have the full context of the system in its working memory, for a codebase of any reasonable size. Any gaps in context when forming abstractions (e.g. through summarization) will lead to a gap in understanding, and thus a gap in reasoning about security. It’s the same challenge that humans face when writing code, and why vulnerabilities exist in the first place.

To find vulnerabilities, on the other hand, is much easier. Perhaps performing novel security research (the kind that takes years, like James Kettle’s work on HTTP request smuggling) is still a bit too much to ask of AI agents just yet, but in reality, most vulnerabilities are repetitions of the same mistakes. This variant analysis is something that AI agents can be good at.

The models will only get more intelligent. At Hacktron, we’ve been consistently impressed by the ability of AI agents to find and exploit vulnerabilities in complex codebases that would have taken us days to fully understand.

I believe that AI will expand the capabilities of threat actors exponentially, at a time when the number of true vulnerabilities in the world is also exploding. Where previously nation-state espionage scaled only with human capital, it will, in a few years, be a far more worthwhile endeavor to simply run a cluster of AI hackers targeting systems that don’t have an army of SOC operators behind them.

The likely targets are tech startups and mid-sized companies, and these are precisely the companies that won’t be prepared for the day this happens. North Korea has so far only been able to afford targeting a handful of carefully selected targets, but this is likely going to change when it becomes much more cost-effective to cast a wider net.

The opportunity

There are two potential futures here:

1. No one except nation-states builds AI hackers, and X years from now, country Y unleashes their AI hackers on the world. This future sucks.

2. Someone builds an AI hacker as a service, the world integrates it into their security stacks, realizes how scary good it is, wakes up, catches up to it, and we are all prepared for the day that country Y unleashes their AI hackers on the world. This future is much better.

The second future is the one that I am betting on. That’s why I believe in Hacktron. Every software company in the world is going to need to be protected round the clock, and the best way to do that is to have a team of AI hackers working for you.

Over the next few years, I believe we will slowly start to see this generational threat unfold.