With the Software Supply Chain, You Can't Secure What You Don't Measure

“You can’t improve what you don’t measure” is an oft-cited bit of wisdom frequently attributed to the famous management consultant Peter Drucker.

It’s hard to dismiss the core truth of the saying, even if — for the record — he didn’t say it, according to the Drucker Institute. After all, metrics from quarterly sales to individual KPIs are the basis for compensation, promotions, and more in 21st century organizations. There is also abundant evidence — from B.F. Skinner, for one — that humans quickly tailor their behaviors to how they’re measured and incentivized.

The question that goes unasked is whether the right things are being measured, or whether the measurements in question are complete enough that they don’t distort the perceptions (and decisions) of those doing the measuring. Those kinds of inconvenient questions informed research sponsored by ReversingLabs, the firm I co-founded, that analyzed six months of reports to the National Vulnerability Database (NVD) maintained by the National Institute of Standards and Technology (NIST).

2022: A Banner Year for CVEs

Our analysis found that vulnerability reports of new Common Vulnerabilities and Exposures (CVEs) to the NVD are accelerating, with 2022 on track to be the biggest year ever for new vulnerability reports. If the current trend holds, there will be more than 24,500 reports by year’s end. That would mark a 22% increase over 2021, which was also a record-breaking year.

Those numbers seem to suggest that software security is deteriorating and that more forceful interventions by the public and private sector are needed to shore up software security. But there’s a lot that’s missed in that quick take. In its two decades of existence, the NVD has seen inconsistent growth in the number of reported vulnerabilities and exposures. Before 2005, the annual number of vulnerabilities assigned a CVE identifier never exceeded 2,500. For more than a decade after that, disclosed issues fluctuated between 4,000 and 8,000 reports per year.

The number of new CVEs in those years reflected the limited capacity of the CVE team at the nonprofit corporation MITRE, which was charged with taking in and documenting new CVE reports. We know that, because once MITRE began inviting more organizations to report vulnerabilities as CVE Number Authorities (CNAs) in 2016, the number of vulnerabilities surged. It doubled in 2017, and each year since has surpassed the previous year’s record of reported CVEs.

The message: Considering a metric like the number of new CVEs is a mostly meaningless exercise if you’re trying to assess the overall state of software security. More contributing firms will result in more CVEs. Fewer contributing firms will produce a drop in CVEs. The overall trend line is meaningless — at least so long as participation in the CVE program and submissions to the NVD by software vendors are voluntary.

Software Supply Chain Security: Unmeasured and Unimproved

As to the question of whether the right things are being measured? Here again, the current configuration of the NVD is misleading and heavily skewed toward the “usual suspects.” Linux distributions Fedora and Debian accounted for 1,123 and 958 vulnerabilities, respectively, in the first half of 2022 and rank first and third on the list of software firms affected by reported issues, our research revealed. Google, Microsoft, Oracle, and Apple accounted for more than 500 vulnerabilities each.

The NVD has far less to say about flaws in popular open source platforms that are getting attention from sophisticated cyber actors. For example, our research shows that attacks on the popular software package repositories NPM and Python Package Index (PyPI) spiked 289% in the past few years, to 1,010 in 2021 from 259 in 2018. But only 56 CVEs referencing PyPI are in the NVD. PyPi’s owner, the Python Software Foundation, is not a CNA. Other popular development and CI/CD platforms like CodeCov, CircleCI, and Bamboo are likewise not CNAs.

Uncomfortable Question: Can We Trust Code?

What does this disconnect mean for the future of public resources like the NVD? Change, for one thing. Recent events — such as the hijacking of the popular ua-parser-js project by a cryptominer — show that even seemingly secure projects can be compromised.

The lesson from incidents like this is that software security teams need to expand their focus beyond vulnerability scanning and even source code analysis to look at what code is doing. As long as we ignore this core question — can we trust code? — we are not addressing software supply chain security.

NIST and the federal government are also slowly pushing forward to implement the White House’s year-old executive order on improving the nation’s cybersecurity, which requires all federal government contractors and software providers to create a software bill of materials (SBOM) that can be reviewed. Recently released practice guidelines from the NSA, CISA, and ODNI further spell out steps organizations can take to secure software supply chains.

To keep pace with these larger changes, however, the NVD also needs to evolve. At the very least, its scope should expand to consistently include software supply chain exposures. Only then will the NVD move closer to representing the full breadth of threats facing modern organizations.