On the Rapid Development of AI in Terms of Vulnerability Discovery
I spoke with the board of FINMA last week. One of the topics we discussed was the impact that artificial intelligence is having on security — the security of businesses, of banks, and of Switzerland as a financial centre. Now that we see vulnerability disclosures based on AI bug discovery at an unprecedent speed.
The National Cyber Security Center recently published a piece putting modern LLMs into perspective. My own outlook, however, is somewhat gloomier — at least in the near term, and possibly the mid term. I have no idea how this will play out over the long run, but let me start with what I can see today.
Open Source as Primary Target
It is obvious that AI capable of identifying bugs will target open source software first — not because open source has more bugs than proprietary software, but because of its open nature. It is simply easiest to analyse code you can see.
But make no mistake: they will come for commercial code too. Decompiling and analysing proprietary binaries is nearly as straightforward for an AI as reading open source. It is somewhat harder for online services, where the code never leaves the server. But one step at a time.
In the near term, open source software is the priority target. Proprietary software installed locally is next.
The Structural Advantage of the Attacker
On the defensive side, AI is powerful enough to be genuinely useful. Open source projects are already adopting it — AI-assisted triage, automated checks, and more. Developers are ramping up their AI game. This is very promising.
Yet there is a structural advantage on the side of the attacker. Attackers need to get lucky only once. They can throw as many exploits at a target as they wish, knowing that sooner or later one will stick, one will work, one will open the gate. This scales effortlessly. The only limit is the money you are willing to spend on AI-driven vulnerability discovery.
Defenders, on the other hand, have to get it perfect. Mere luck with one exploit is not enough — the next attack will find them soon enough.
The Human Bottleneck
One thing we have learned about AI is that it can be remarkably good, but it is not always consistent — and it is certainly not perfect. The level of perfection needed in source code and systems can ultimately only be achieved with a human in the loop: human code review that assesses the work of the AI.
Open source developers capable of reviewing code at the required depth are rare. It takes expert-level skill to be a good reviewer, and this resource has always been scarce. Now, with AI amplifying the pressure on open source projects — more security bugs being identified and reported, more pull requests from unknown origins being submitted — these scarce review resources are needed more than ever.
Meanwhile, an attacker can scale with ease. That means attackers hold the advantage, at least in the near term — and possibly the mid term as well.
And there is the increasing danger of malicious contributors, human or AI, disguising as supporter of a project, that is struggling to keep up and they make themselves useful until they obtain commit privileges.
The Open Source Funding Crisis
There is an additional dimension I shared with the FINMA board. Open source developers overwhelmingly operate on a shoestring budget. For security-focused projects, fixing security bugs as they arrive is an intrinsic motivation. But for feature-oriented projects, security bugs are a nuisance. The desire to fix reported vulnerabilities in a timely and high-quality manner wears thin when developers receive no support from the wider community or from the commercial integrators profiting from their work.
This is a security problem. Open source software can become a liability when the pressure grows so intense that developers step away from their projects. Security bugs will go unfixed, and the commercial integrators — based on my experience — are likely the last ones to realise that their supply chain is no longer being maintained.
The market is not fixing this. The problem has been known for years, but it is intensifying now. There is a chance that the European Union’s Cyber Resilience Act will force commercial integrators to stop being free riders on their open source supply chain. They will only be able to guarantee the security of their commercial offerings when they actively support the open source components they depend on. I’m not sure how this will play out, namely in Switzerland. But the status quo is neither sustainable nor secure.
Every New Model, a New Challenge
A final thought. The new LLMs coming out are capable of identifying security bugs in code, and with every release, new capabilities emerge. Every new model is more advanced in the discovery of vulnerabilities.
That means tomorrow a new model may be released that can discover a whole new class of security bugs across your entire codebase. In consequence, every codebase would have to be re-examined with every significant new LLM release and every vulnerability identified would have to be fixed in production before the malicious players with access to your code or service are able to use an exploit that the AI developed based on the same discovery.
I do not yet see how we could solve that problem.