As probably so many of you, I’m currently in various discussions on the dawn of vulnerabilities discovered by AI. While the topic isn’t new, the dimensions very well might. Here a few thoughts.

Worst-Case Scenario

Not wanting to say, “oh my god, we’re all gonna die”, just to paint a certain mood: Finding vulnerabilities has become easier using various AI solutions, this shouldn’t be a discussion point anymore, but a fact. To stress here, I’m not saying the quality of everything it produces is high, but it does definitely have its moments!
Let’s just imagine things develop further into this direction, prices could drop a little, tools get more and more available and thus the frequency with which AI found vulnerabilities grows. Additionally skilled researchers utilize AI to scale and automate, just as when fuzzing came out as a widely available method.

So, our worst-case scenario we have: Our monthly patch-day turns into a weekly patch-day, turns into a daily patch-day. Otherwise said: Ops is under a fulltime DDoS.

Patching Probably isn’t even our Problem

Let’s say we have widely automated infrastructure, thus installing patches is “quick”, we still have that pain point challenge …ooooooooof… testing! The past has shown us, that many vendors, while fixing a specific issue or while adding a certain feature often change more than they disclose. As such we have learned to not blindly trust patches, due to them potentially causing issues in our very own environment.
History has brought us all to pretty long and complex change management processes, which while being required, no questioning here, do create a notable overhead, which for a daily patch-day will, please excuse my French, f*** us all completely.
And >no< not testing and skipping change processes should not be the solution

A Proposition

I actually planned to write this section a little differently. While working on it, I thought about Microsoft’s Knowledge Base articles and related patches, which I suppose are where I was heading.

Please create small, tiny, modular and isolated patches for issues, especially for appliances.

At the moment, patches for appliances and larger software always contain various fixes and changes in one blob. While being quick to install and allowing all-in-one testing, especially on the manufacturer’s side, on the operator’s side this brings a significant amount of complexity: Even if I just need to patch a specific CVE, I usually have multiple other things I update at the same time and thus more things I need to test before patching in prod than necessary.
In an easier scenario, I log in and don’t see one version update, but maybe 2..3..4..8 different tiny patches, each one for a specific component. Thus, my emergency change solely covers the part that is actually creating the emergency, the other parts I patch in my planned patch window.

Continuous Patching

There are a few calls for and dreams about dropping the “patch window” approach and continuously patching, which I would also appreciate a lot. Sadly, I have the feeling this approach comes from a misconception: In software development, where this is heavily done, each change is tiny, often isolated and as such “quickly” tested. Even if we’re talking about multiple changes in prod, each one is individually tested and evaluated in a lower environment to ensure prod carries on running smoothly. Thus, here we’re kind of living my proposal. When talking appliances on the other hand, we as “users” do not have access to the lower environments, as they belong to the manufacturer. Thus, we only get to test the combined change containing all tiny things….

A Lack of Trust

Things can go wrong, things will go wrong, this is a part of life. Sadly, we have collected so much bad experience, that strict and lengthy change management processes have become a necessity. As such, us not patching quickly today simply results from a lack of trust, that was established over millions of patches.
As rebuilding that trust will take a long time, maybe the technical approach would be a good start?

Tiny Patches -> Tiny Risks -> Quick Testing -> Quick Rollout -> Quicker Security