Patch management needs a revolution, part 4: Sane patching is safe patching is selective patching

This is the fourth part of Vincent Danen’s “Patch management needs a revolution” series.

• Patch management needs a revolution, part 1: Surveying cybersecurity’s lineage
• Patch management needs a revolution, part 2: The flood of vulnerabilities
• Patch management needs a revolution, part 3: Vulnerability scores and the concept of trust

One of the biggest concerns with modern patch management is that we haven’t truly challenged our thinking around “patching everything” over the past 40 years. Today, we are still inundated with customer requests to patch everything, despite the available evidence showing that most vulnerabilities do not and will not ever see exploitation. So the larger question is, what is the purpose of the exercise? Is it to have vulnerability-free software (a compliance-driven exercise) or is it to minimize the potential or severity of a breach when it occurs (a risk-driven exercise)?

If the goal is to reduce the number of breaches, then this single-minded focus on software simply does not make sense. Yet, most corporations and regulators focus exclusively on the software element and do not pay near enough attention to the human element.

If we look at the 2023 Verizon Data Breach Investigations Report, as one example, it clearly indicates that 74% of breaches are attributable to the “human element” (stolen credentials, phishing, weak passwords, misconfigurations, etc). Exploited software that resulted in a breach accounted for a mere 5% of breaches, less than 2022’s 7%. This evidence tells us that the vast majority of breaches are done without software exploitation. One might ask the question of whether we have made a dent in reducing breaches at all, considering that every year breach numbers continue to rise.

Yet we focus, almost exclusively, on the old way of thinking: that we must patch everything to eliminate breaches. At best, patching everything immediately will potentially reduce the number of breaches by 5%. But even this assertion requires further analysis, because it wasn’t all 25,000 CVEs in 2022 that were exploited. So while fixing 25,000 CVEs might reduce 5% of the breaches, it’s also probable that, given the evidence, fixing 1,000 of those 25,000 CVEs could also reduce 5% of breaches. Fixing the other 24,000 could at best do nothing and be a useless expenditure of time and resources or, at worst, contribute to other costly factors such as new bugs that need to be fixed, new vulnerabilities that are introduced, not to mention the time and effort required to test and apply all of those patches.

What is also not clear is whether those breaches due to software exploitation were because patched software was unavailable or because it was unapplied. There is a distinct difference between the two. In other words, it’s also possible that patches existed and were not applied quickly enough, resulting in a breach. That could mean that the current volume of vendor-supplied patches is sufficient and that creating more patches gains nothing.

Let’s use an illustration to clarify, based on Red Hat software and known exploitation rates for vulnerabilities in that software in 2022. Let’s assume that it costs each end-user $1000 to deploy a patch, any patch. This is a wildly imprecise but simple example. Every organization will adjust this figure upwards based on testing, deployment and scale, among other factors. If we wanted to definitively patch everything, knowing that a small percentage of companies are breached annually, and we wanted to focus on that 5%, given the number of vulnerabilities the KEV found exploited in the wild in 2022, we would be investing $7,000 to fix 7 CVEs (the number exploited and reported in the 2022 Red Hat Product Security Risk Report). That would reduce our risk of a compromise leading to a breach through software substantially, given those vulnerabilities are being actively exploited. (As a quick summary to the numbers we’ll be using, taken from the 2022 Risk Report, there were 1,656 total vulnerabilities, 19 were Critical, 276 were Important, 1,086 were rated Moderate and 275 were Low. Only 7 were known to be actively exploited as per CISA’s KEV (Known Exploited Vulnerabilities).)

But, you don’t know which vulnerabilities will be exploited so you want to pre-emptively fix them all because maybe one of those other 1,656 CVEs might be the one that an attacker exploits. Assuming we had fixes for them all, and again assuming the same cost, that $7,000 investment balloons to $1,656,000. $1.6M to reduce the risk of a breach by 5% is a very expensive proposition. Is that the wisest course? Using the same model, fixing all of the Critical and Important issues, which are those most likely to result in a breach, would require fixing 295 vulnerabilities at a cost of $295,000 as per our model.

This definitively highlights the rationale of focusing on those high impact vulnerabilities as a reasonable cost. For example, we’ll consider the “cost to avoid” for each severity type. As per Red Hat’s 2022 data, 19 Critical CVEs fixed to avoid two exploitable vulnerabilities nets a “cost to avoid” of $9,500 per exploit. The cost to avoid three exploited Important issues, by fixing all 276, nets a “cost to avoid” of $92,000 per exploit. These are reasonable costs to avoid a breach.

Of the remaining 276 Low vulnerabilities, $276,000 is spent avoiding nothing as none were exploited, and the cost to avoid two exploited Moderate issues, out of 1,086, is a staggering $680,500 per exploit. Given an exploited Moderate issue rarely yields meaningful damage, these are very expensive costs to avoid something that is either difficult to exploit to begin with, or does not give an attacker sufficient access to cause significant damage. Let’s also keep in mind that attackers focus on a good return on investment as well, so they target higher impact vulnerabilities that can cause the most damage.

An illustration I use when talking on this topic might be helpful.

There is a further complication to be aware of. CVEs are not necessarily one-for-one and with the ubiquity of open source, one component might be used in multiple products. We saw this with the Log4j issues in 2021 – the Log4j component was found in many applications, meaning one CVE could be present in dozens of applications. That means the real cost to fix dramatically increases based on the amount of software an organization uses. Again, $1,000 per CVE is illustrative; the real cost is substantially higher.

By focusing on the vulnerabilities that matter, while still expensive, more than $1.3M is saved. This savings could be invested in the area that contributed to 74% of breaches, as per Verizon’s report: the human element. One easy target for that investment is technologies to prevent the misconfiguration of software. Managing configurations of software is a large part of the challenge, arguably easier to solve than human behavior. Red Hat offers tools that can be used to help, including Red Hat Ansible Automation Platform, Red Hat Advanced Cluster Security for Kubernetes and the Compliance Operator available on Red Hat OpenShift Container Platform.

This is where a risk-based approach makes the most sense and is most cost-effective. Particularly today, as corporations are struggling to grow while remaining profitable and competitive, a capital expenditure on fixing things that don’t need to be fixed is an expensive luxury proposition, or the most expensive insurance policy available. Added to that, it just doesn’t work, nor does it provide the expected value or protection. If the goal is to reduce risk, corporations must focus on people and process as much as, if not more than, on technology. Of course, it’s difficult to provide a comprehensive framework that is suitable for all companies, so it is imperative that corporations evaluate their own risk tolerance and where the most valuable return for their investments can be found.

It’s 2024 and we’re stuck in ‘90’s thinking.

I’ve spent these four posts mostly laying out problems of varying degrees without a solution. As with anything, there’s not a pure silver bullet for evolving how we, as an industry, view patching. But, we can certainly find ways to improve and become more effective without actually weakening our cybersecurity postures. Here’s a hint: It involves open source.

Vincent Danen lives in Canada and is the Vice President of Product Security at Red Hat. He joined Red Hat in 2009 and has been working in the security field, specifically around Linux, operating security and vulnerability management, for over 20 years.

Read full bio