Mourning the Margins

Welcome to possibly my one and only blog post on my new blog, this started as a Signal message but got too long for an LinkedIn post. It's about the new AI found vulnerabilities, but I want to talk about how it interacts with the reliability aspect.

I think the security margins that we have gotten used to over the last 20 years will go away, and it will be painful to adjust. The solutions are in some ways new, but in other ways 20-30 years old.

A bit of background reading

Gadi Evron put together a summary that I'm just going to modify a bit.

2025:
In June we got the first inkling, Rob Royce from NSAs Tailored Access Operations (read: hackers) said AI will soon be a great exploit coder.
In October three senior security people posted Autonomous AI hacking and the future of cybersecurity.
In November Anthropic posted about Chinese state-sponsored attackers using Claude in real attacks by claiming it's part of a CTF or by splitting apart tasks so it didn't get the full picture.

2026:
In February Phil Venables, various CISO roles etc under his belt wrote Things Are Getting Wild: Re-Tool Everything for Speed
In March Nicholas Carlini stood on stage at unprompted and held his talk about Anthropics new model finding vulnerabilities without any advanced prompting.
This led to Thomas Ptacek posting Vulnerability Research Is Cooked.
In April Detectify posted about how the breakout time is now down to 29 minutes, related to the APTs Anthropic mentioned.

And in case anyone missed it, Anthropic is holding onto the latest model, "Mythos", and is starting Project Glasswing where they help selected companies and projects fix their vulnerabilities. But this is only buying a bit of time, it's not a solution. Open weights models that people can download and run wherever are 3-12 months behind the frontier models.

Give them a read if you haven't, then if you're coming into this with fresh eyes go brew yourself a nice cup of tea to calm your nerves.

Some examples of the impact

Open source projects are already facing a quickly growing inflow of vulnerabilities already. E.g the Linux kernel have gone from 2 per week to 5-10 per day. And these are good proper vulnerabilities, these aren't the AI slop vulnerabilities of a year ago. They're also seeing duplicate reports for the first time ever. The creator of curl is a seeing similar influx.

HackerOne has paused it's open source bug bounty programme. Source, readable article.

Open Source

The next few years will not be much fun for open source. Anthropic holding onto Mythos buys a bit of time but not much, considering how many reports projects are already facing.

ChainGuard posted an article with a rather spicy title Open source died in March. It just doesn't know it yet. Do note that their whole business model is selling secure open source, so it's clearly angled but it's still another data point in this growing dark cloud.

We will end up with much better software at the other end, but also many more burnt out maintainers which means abandoned projects. Even before this flood wave of vulnerabilities they were in the vast majority of cases unpaid and unthanked. The classic XKCD.

Changing environment

So what do we have?

Agentic attackers working on machine speed, those 29 minutes breakout time will go down, not up.
A flood wave of vulnerabilities that open source maintainers are unlikely to be able to handle. The linux kernel has already gotten duplicate reports, odds are if you have found a vulnerability someone else has too. Embargoes doesn't really work anymore. They buy a bit of time, how much? unclear.

So, what do to? "We'll patch faster!" is the response from some people.

What if there's no patch yet? See above about unthanked, unpaid and burnt out maintainers. Will you replace all your open source with in house code? With agents it's certainly possible. But you will multiply your code base & build times & CI costs for the same functionality, is it worth it?

The Reliability angle

I've seen loads written about the above by people much more knowledgeable than me. What I haven't seen yet is connecting to reliability, and I think they're on opposite sides of this and unifying them will take work. Possibly a lot. The good thing is you'll end up with way faster and solid processes and tooling.

What if it's a bad patch that breaks things because they don't have 100% test coverage in their "oh this could be a fun weekend project" that now your company relies on.

To start with, how do you patch? Do you have mutable VMs and patch live? Do you auto-deploy VMs from images according to the phrase "cattle, not pets"? Or go serverless and just have functions called on a rotating pool of VMs?

Maybe so far you have done weekly releases to prod if you're a halfway decent SaaS company. Recreating a VM once a week with some blue-green swap over isn't a problem. This means that have a maximum patch window of one week. What happens if that patch window needs to shrink to 30 minutes? 20 minutes if you want to stay within that 29 minute breakout time. You'll be spending more on deployments than on actually serving customer traffic.

Patching everything at once leads to risk. You need a staging environment to check for bad patches, how long do you want patches to soak in staging? Overnight? That's 12 hours.

After staging you probably want a gradual rollout in case it's something missed in staging, how long? Another 12 hours until a release reaches the whole fleet? Or maybe you have your early customers that get releases first in exchange for lower cost? If you look at the GKE release schedule version 1.30 was available in Rapid 2024-04-30, Rapid is their "We don't make any promises about reliability, there's no SLA". It reached Regular, which is their recommended version, on 2024-07-30. That's three months of testing, validation and making sure it's ready for wide scale usage. Because people build their entire companies on GKE, if they fail things go very badly. I don't know how to cut those 3 months of validation to 1 month without just skipping 2/3 of it, or to 1 week, or to 1 day. But I think we might need to figure it out.

Not patching leads to security risk. Patching leads to reliability risk. Which is worse? How much of each is okay? Only you can decide that.

What if you don't have a SaaS with a fleet of nodes but produce client side software, or in rare cases an operating system. Some software can't crash because peoples companies and livelihoods depend on them. You can't deploy a new operating system version every 30 minutes.

And on the subject of operating systems. LTS, "Long Term Support", versions of distributions and software have been a stable in IT environments until now where the maintaining organization promises to provide support for 5 years, 10 if you pay extra. Backporting security fixes for the linux kernel for a 6 month old LTS version isn't that bad. A 4½ year old is very likely quite a lot worse. A 9½ year old version sounds like a nightmare. A person I know got root on an K8S node running an OS image with an out-of-memory vulnerability from July 2025. The node is running the most recent available image of that OS. With a 9 month vulnerability.

I'm not sure free LTS versions will survive this. The 5 year version might become paid, and the 10 year version might get much more expensive.

Looking back

For the last 20 years we've had it kind of easy. As long as you weren't in defence or banking you didn't need to be really strict about security. If you got breached you posted a "We're so sorry" blog post and 6 months later everyone had forgotten about it. Well, almost. Unless you really, really mess up.

It was okay to have patch Tuesday once a month. It was okay to have "passwort!" as the admin password. Or just admin. It was okay if your php server used raw sql commands without prepare statements. It was okay if your cgi-bin parser used strcpy without properly checking length and delimiters.

Things rarely got that bad, because there were mostly very little money in it. This began to change with the advent of various pseudo-anonymous cryptocurrencies. Previously if you wanted $20M in ransom from a company they needed to wire you that money, and then the bank could react. If you wanted cash they knew where you and/or a companion of yours would be. Getting paid wasn't possible, but it was a huge faff and almost nobody bothered.

With cryptocurrencies you could sit in your house in Russia and crime to your hearts content, get them to pay you in bitcoin and no bank can claw it bank. Send a part of it to your FSB handler to make sure the police doesn't interfere and it's gravy. Maybe buy another ferrari on the way home?

But it still wasn't that bad, it was much much worse, but not that bad. Companies had to improve, but marginally. Most didn't, and it was okay. Exemplified by the number of companies still getting breached by playing silly buggers with their security setup.

Looking forward

I think this will change, and here we come to the title of this post. The loss of margins.

I think the margins will go away. The margin from "I'm secure" to "time to panic" will shrink, massively.

Vulnerabilities exist, and now much much more people will know about them. Some will let maintainers know, others will use them.

Agentic attackers means that if they're in they move, fast. If you have 29 minutes to detect them and your logging & detection & alerting system takes 15 of them, you have 14 minutes to wake up, get situated and decide whether it's worth taking down production or not.

You won't have time to do a manual threat modelling once a year.
You won't have time to patch every month. Or week.
You won't have time to react when you get a breach alert.
You won't have time.

If you have nation state hackers after your systems 24/7 you don't have much choice or the next fighter jet schematics are being read in China before you can blink. Now the margins between those companies and "normal" companies will need to shrink. Or the normal companies will not have a good time.

All of this is new, and the security business is trying to find it's footing again. Assumptions that were valid 20 or 10 or 5 or even 2 years ago are now crumbling.

So what to do?

Good news #1: You don't need to buy any new fancy AI security tooling for $500 / seat / month.

Good news #2: your CISO has known about these things for the last 20 years.

Bad news: This will cost time. Possibly quite a bit.

If you assume breach, how do you handle that? This is similar to how people Should(TM) handle successful phishing attacks, not by human action but by being prepared.

Data classification. What data is really sensitive? If something is breached, what parts means it's a crisis and what parts means it's a problem? Maybe it's you won't go bankrupt if Bobs little QA thingy starts mining cryptocurrency.
Remember about patch times? When you have your data classification done you have your priority levels for patching. Anything that can reach the critical things gets patched first. Then the things that can reach those things.
What you end up with here is Attack Path Analysis / Attack Path Simulation, it's an optimization problem where you want to increase organization security as much as possible by each patched systems. What's on the critical path? Which are the chokepoints between entrypoints and your high value assets?
Least privileges. Enforce them. Everywhere. If an attacker doesn't get any permissions they can't do as much. Yes, it will impact your staff. No, they can't do everything they used to do.
Network segmentation. If the attacker can't move freely between machines they can't do as much. Network segmentation is the the most common way to limit blast radius. Pick a point in your environment and ask yourself "How much else can go badly if someone gets <here>?". You want to shrink that circle as much as possible.
Credential rotation. Don't let users / keys / certificates remain if they're unused for too long. Review and validate regularly.
Immutability. The fewer things that can change the better. Operating system, configuration files etc.
Limit data at endpoints. Maybe look into setting up a hosted environment for your developers to connect to from their laptops. If your endpoint gets phished they don't get as much. Maybe swap from desktop office to a cloud office system. Personally this is the one I'm the most against because so far no hosted dev environment has been good enough...
Connected to the above, if you have no data only on employee laptops, that means you can wipe them when there's sign of a breach. And then just re-image then when needed.
Security Champions. Set up a Security Champions programme to raise the average level of your teams. Gamify it with buttons or special T-shirts or other fun stuff so it's something nice and not just extra work for those really interested.

AI can't solve all of this for you, but it can help you get part of the way there and it can build tooling for a lot.

The cost of building custom tooling has dropped like a rock the last year. Some of it can't be automated, e.g. data classification, but you can build a tool doing least privilege checks and check time between usage and now, produce a list and send out to relevant people. First go read the words of an AWS principal engineer. You can now build custom tooling to detect and enforce these things continuously. You can build the fancy pipelines to run tests in parallel over different machines to validate patches faster. You can build layered system tests, maybe all of them don't need to run for every patch. Your engineers probably have loads of even better ideas here. The goal is time, can you go from validating a patch in a week to one day? one hour? What would that take?

Agentic defence

AI can help you speed things up in other ways. Agents that do log analysis, anomaly detection, configuration drift detection, investigations, everything that saves you some of the remaining 14 minutes out of the 29 once an alert has been filed.

But if you're willing to go beyond read-only, what actions are you willing you let your agents take? If a possible breach is detected, would you let your agent disconnect that machine? Would you disconnect it at 90% likelyhood? 80%? 50%? And as we know all LLMs hallucinate, what's the risk / benefit analysis for your case?

A friend trial-read this blog post and had an analogy. When Spotify released their now quite famous "This is how we do software development at scale" with Tribes etc a bunch of companies tried to copy that but had...variable...success, because they didn't have the structured experimentation and frameworks in place to apply it. His point, that I think I agree on, is that we need experimentation, and the experience that comes from experimentation. The context for your organization and your IT environments will matter, and that context is something you need to build.

We've seen a lot of sandboxes for agents in development, some local and some hosted etc but so far I haven't seen many similar things for defensive agents. If you have the LLM analysis result in "this machine looks compromised" and you have a custom tool checking "that's a sales rep machine" in the inventory system followed by a calendar check "they don't have a sales meeting now or in the next hour" it can possibly safely be disconnected and a text message sent to their phone that they need to get a loaner laptop and have it re-imaged.

Agents are great at finding patterns, we just need to figure out how to best to apply them. Attackers don't care nearly as much. If they torch half of production they can very likely just move on to the next target. Defenders needs to balance speed, reliability and many other aspects, while matching the attackers speed.

Looking ahead

I think we'll look back at the previous 20 years with kind eyes and smile to ourselves how easy we had it, how "silly things like that would never fly these days" etc.

We're in for a rough couple of years and there will be some pain, but overall I'm hopeful and I think that software engineering as a whole will come out stronger. Because it's either that or we just stop. And we're still needed, software is a key part of the modern world.

We just have to figure out how to do this securely, reliably, quickly, at scale.