XZ Utils Scare Exposes Hard Truths About Software Security

Source: Juliana_haris via Shutterstock

The recent discovery of a backdoor in the XZ Utils data compression utility — present in nearly all major Linux distributions — is a stark reminder that organizations who consume open source components ultimately own responsibility for securing the software.

XZ Utils, like thousands of other open source projects, is volunteer-run and, in its case, has a single maintainer managing it. Such projects often have little to no resources for handling security issues, meaning organizations use the software at their own risk. That means security and development teams must implement measures for managing open source risk the same way they do with internally developed code, security experts say.

“While it’s unlikely an organization can effectively prevent [all] exposure to supply chain risks, organizations can absolutely focus on a strategy to reduce the probability that a supply chain attack is successful,” says Jamie Scott, founding product manager at Endor Labs.

Open source is not the same as outsourcing: “Open source maintainers of software are volunteers. At an industry level, we need to treat them as such. We own our software; we are responsible for the software we re-use.”

Well-Intentioned, Under-Resourced

Concerns over open source software security are by no means new. But it often takes discoveries like the Log4Shell vulnerability and the backdoor in XZ Utils to really drive home just how vulnerable organizations are to components in their code. And often, the code comes from well-intentioned yet hopelessly under-resourced open source projects that are minimally maintained.

XZ Utils, for instance, is essentially a one-person project. Another individual managed to sneak the backdoor into the utility over a nearly three-year period, by gradually gaining enough trust from the project maintainer. If a Microsoft developer had not chanced upon it in late March when investigating odd behavior associated with a Debian installation, the backdoor might well have ended up on millions of devices globally — including those belonging to large corporations and government agencies. As it turned out, the backdoor had minimal impact because it affected versions of XZ Utils that were only present in unstable and beta versions of Debian, Fedora, Kali, open SUSE, and Arch Linux.

The next such open source code compromise could be far worse. “The scariest part for enterprise organizations is that their applications are built on top of open source software projects just like XZ Utils,” says Donald Fischer, co-founder and CEO of Tidelift. “XZ Utils is one package of tens of thousands that are in use every day by typical enterprise organizations,” he says.

Most of these organizations lack sufficient visibility into the security and resilience of this part of their software supply chain to be able to evaluate risk, he notes.

A recent Harvard Business School study estimated the demand-side value of open source software to be an astonishing $8.8 trillion. Maintainers are at the core of this ecosystem and many of them are flying solo, Fischer says. A survey conducted by Tidelift last year found 44% of open source project maintainers describe themselves as the sole maintainers of their projects. Sixty percent identified themselves as unpaid hobbyists, and the same percentage said they have either quit or have considered quitting their roles as project maintainers. Many maintainers described their efforts as stressful, lonely, and financially unrewarding work, Fischer says.

“The XZ utils hack brings into stark relief the risks of under-investing in the health and resilience of the open source software supply chain [that] enterprise organizations rely on,” Fischer says. “Enterprise organizations need to realize that the majority of the most relied-upon open source packages are maintained by volunteers who describe themselves as unpaid hobbyists. These maintainers are not enterprise suppliers but are expected to work and deliver like them.”

Danger: Transitive Dependencies

A study that Endor conducted in 2022 found that 95% of open source vulnerabilities are present in so-called transitive dependencies, or secondary open source packages or libraries that a primary open-source package might depend upon. Often, these are packages that developers don’t directly select themselves but are automatically employed by an open source package in their development project.

“For example, when you trust one Maven package, on average there are an additional 14 dependencies you implicitly trust as a result,” Scott says. “This number is even larger in certain software ecosystems such as NPM where you on average import 77 other software components for every one you trust.”

One way to start start mitigating open source risks is to pay attention to these dependencies and be selective about what projects you choose, he says.

Organizations should vet dependencies, especially the smaller, one-off-packages, manned by one- and two-person teams, adds Dimitri Stiliadis, Endor’s CTO and co-founder. They should determine if dependencies in their environment have proper security controls or if a single individual commits all code; whether they have binary files in their repositories that no one knows about; or even if someone is actively maintaining the project at all, Stiliadis says.

“Focus your efforts on improving your response effectiveness — foundational controls such as maintaining a mature software inventory remains one of the highest value programs you can have in place to quickly identify, scope, and respond to software risks once they’re identified,” Scott advises.

Software-composition analysis tools, vulnerability scanners, EDR/XDR systems, and SBOMs can also all help organizations quickly identify vulnerable and compromised open source components.

Acknowledging the Threat

“Mitigating exposure starts with shared understanding and acknowledgement in the C-suite and even at the board level that roughly 70% of the ingredients of the average software product are open source software historically created by mostly uncompensated contributors,” Tidelift’s Fischer says.

New regulations and guidelines in the financial services industry, the FDA, and NIST will shape how software is developed in the years ahead and organizations need to prepare for them now. “Winners here will quickly adapt from a reactive strategy to a proactive strategy to managing open source-related risk,” he says.

Fischer recommends that organizations get their security and engineering teams to identify how new open source components come into their environment. They should also define roles for monitoring these components and proactively remove ones that don’t fit the company’s risk appetite. “Reacting to late stage problems has become an ineffective way to deal with the scale of the risk to the business over the last several years, and the US Government is signaling that era is coming to an end,” he says.

About the Author(s)

Jai Vijayan is a seasoned technology reporter with over 20 years of experience in IT trade journalism. He was most recently a Senior Editor at Computerworld, where he covered information security and data privacy issues for the publication. Over the course of his 20-year career at Computerworld, Jai also covered a variety of other technology topics, including big data, Hadoop, Internet of Things, e-voting, and data analytics. Prior to Computerworld, Jai covered technology issues for The Economic Times in Bangalore, India. Jai has a Master’s degree in Statistics and lives in Naperville, Ill.