XZ Utils Backdoor Implanted in Carefully Executed, Multiyear Supply Chain Attack

Source: ozrimoz via Shutterstock

A newly discovered backdoor in XZ Utils, a data compression utility present in nearly all Linux distributions, has revived the ghosts of previous major software-supply chain security scares such as the Log4Shell vulnerability and the attack on SolarWinds.

The backdoor is embedded in an XZ library called liblzma and gives remote attackers a way to bypass secure shell (sshd) authentication and then gain complete access to an affected system. An individual with maintainer-level access to the code appears to have deliberately introduced the backdoor in a painstakingly executed, multiyear attack.

A Shock to the OSS Community

The backdoor affects XZ Utils 5.6.0 and 5.6.1, which are versions of the utility currently used only in unstable and beta releases of Fedora, Debian, Kali, open SUSE, and Arch Linux. As a result, the potential threat with this backdoor for now is considerably more limited than if the malware had found its way into a stable Linux distro.

Even so, the fact that someone managed to sneak a nearly undetectable backdoor into a trusted, widely used open source component — and the potential havoc it could have caused — has come as a painful wakeup call on how vulnerable organizations remain to attacks via the supply chain.

“This supply chain attack came as a shock to the OSS community, as XZ Utils was considered a trusted and scrutinized project,” JFrog researchers said in a blog post. “The attacker built up a credible reputation as an OSS developer over the span of multiple years and used highly obfuscated code in order to evade detection by code reviews.”

XZ Util is a command-line utility for compressing and decompressing data in Linux and other Unix-like operating systems. Microsoft developer Andres Freund discovered the backdoor in the software when investigating odd behavior in recent weeks around liblzma on some Debian installations. After initially thinking the backdoor was purely a Debian problem, Freund discovered the issue actually impacted the upstream XZ repository and associated tarballs or archive files. He publicly disclosed the threat on March 29.

Over the weekend, security teams associated with Fedora, Debian, openSUSE, Kali, and Arch issued urgent advisories alerting organizations running the affected Linux releases to immediately revert to earlier, more stable releases of their software to mitigate the potential risk of remote-code execution.

Maximum Severity Vuln

Red Hat, the primary sponsor and contributor to Fedora, assigned the backdoor a vulnerability identifier (CVE-2024-3094) and assessed it as a maximum severity risk (CVSS score of 10) to draw attention to the threat. The US Cybersecurity and Infrastructure Security Agency (CISA) joined the chorus of voices urging organizations using affected Linux distributions to downgrade their XZ Utils to an earlier version, and to hunt for any potential activity related to the backdoor and report any such findings to the agency.

All of the advisories offered tips for users on how to quickly check for the presence of the back-doored XZ versions in their code. Red Hat released an update that reverts XZ to previous versions, which the company will make available via its normal update process. But users concerned about potential attacks can force the update if they don’t want to wait for the update to become available via the normal process, the company said.

Today Binarly released a free tool that organizations can use to look for backdoored XZs as well.

“Had this malicious code been introduced to stable OS releases in multiple Linux distributions, we could have seen in-the-wild exploitation en-masse,” says Scott Caveza, staff research engineer at Tenable. “The longer this went unnoticed, the greater the potential for more malicious code from whomever this malicious actor might be.”

In an FAQ, Tenable described the backdoor as modifying functions within liblzma in such a way as to allow attackers to intercept and modify data within the library. “In the example observed by Freund, under certain conditions, this backdoor could allow a malicious actor to ‘break sshd authentication,’ allowing the attacker to gain access to an affected system,” noted the researchers.

XZ Utils “Maintainer” Behind the Backdoor

What makes the backdoor especially troublesome is the fact that someone using an account belonging to a maintainer of XZ Util embedded the malware in the package in what appears to have been a carefully planned multiyear operation. In a widely referenced blog post, security researcher Evan Boehs traced the malicious activity back to 2021 when an individual using the name Jia Tan created a GitHub account and almost immediately started making suspicious changes to some open source projects.

The blog post provides a detailed timeline of the steps Jia Tan and a couple of other individuals took to gradually build enough trust within the XZ community to make changes to the software and eventually introduce the backdoor.

“All the evidence points to social manipulation being used by a person with the sole end goal of inserting a backdoor,” Boehs tells Dark Reading. “Basically, there was never a genuine effort to maintain the project, only to gain enough trust to insert [the backdoor] quietly.”

Typically, gaining commit access to a repository requires an individual to establish a sense of trustworthiness. Often, projects give new commit access to individuals only when there is a need for it and after some risk assessment, Boehs says.

“In this case, Jia created a [seemingly] legitimate need for more maintainers … and then began building trust. Our society is built on trust, and occasionally some crafty people exploit it,” he notes. “Gaining permission requires trust. Trust takes time to establish. Jia was in it for the long game.”

Boehs says it’s unclear when exactly Jia Tan became a trusted member of the repository. But soon after his first commit in 2022, Jia Tan became a regular contributor and is currently the second-most active on the project. GitHub has since suspended Jia Tan’s account.

Saumitra Das, vice president of engineering at Qualys, says what happened with XZ Util can happen elsewhere.

“Many critical libraries in open source are being maintained by volunteers in the community who are not paid for it and can be under pressure due to their personal issues,” Das says.

Maintainers who are under pressure often welcome contributors who are willing to spend even a little bit of time on their projects. “Over, time, such folks can gain more control over the code,” as was the case with XZ Utils, he says.

About the Author(s)

Jai Vijayan is a seasoned technology reporter with over 20 years of experience in IT trade journalism. He was most recently a Senior Editor at Computerworld, where he covered information security and data privacy issues for the publication. Over the course of his 20-year career at Computerworld, Jai also covered a variety of other technology topics, including big data, Hadoop, Internet of Things, e-voting, and data analytics. Prior to Computerworld, Jai covered technology issues for The Economic Times in Bangalore, India. Jai has a Master’s degree in Statistics and lives in Naperville, Ill.