Understanding Red Hat’s response to the XZ security incident

March 29, 2024 is a day that will hardly be forgotten by the open source community: Andres Freund disclosed his findings about the compromise in the xz compression library, which would enable an attacker to silently gain access to a targeted affected system. How did that coordination work under the hood? In this article we will give a behind the scenes glimpse into what this looked like at Red Hat.

****Discovery****

On Wednesday, March 27, Andres contacted the Debian security team via their contact email ([email protected]) and let them know about the oddities he found in a SSH slowdown when using a new XZ library that was shipped by Debian.

The Debian security team relayed this information to the Red Hat Information Security (InfoSec) team. Our InfoSec team is responsible for protecting our network perimeter and company assets.

****Understanding the threat****

InfoSec involved Red Hat’s Product Security (ProdSec) department, the ones in charge of securing the software that we provide to our customers and community. Then, this global multidisciplinary team containing many of our finest developers, PenTesters, offensive analysts, managers, directors and program managers started the then confidential efforts to properly establish, understand and assess the findings from Andres.

After establishing the gravity of the situation, InfoSec and ProdSec engaged their Incident Response Plans (IRP), sharing a more complete view of this embargoed incident to help further understand how it works and assess any potential compromise to our products, both upstream and especially in Red Hat Enterprise Linux (RHEL), as well Red Hat internal assets.

At this stage, there were two major fronts: The most urgent was to determine to what extent, if any, the malware had infiltrated our products. The second was to analyze the malicious nature of the package and better understand its cleverly concealed malicious payloads.

****Securing the fort****

Our next step was establishing the affectedness of our systems, as this presented a number of concerns: Were build systems compromised? Were any other external or internal systems affected? Was there any data exfiltration? If so, was a footprint left behind?

Over the course of the investigation, we found that this is a kind of dormant malware targeting a very specific set of systems: Specifically the x86_64 architecture, requiring both systemd and sshd. It was not targeting BSDs, Android cell phones, wi-fi routers, IoT devices, Raspberry Pis, or anything else. Essentially, it had to be a standard Linux server.

Querying VirusTotal for the rogue .o object file hash returned no results, so it was indeed something not widely known prior to Andres’ finding.

At this point, we established that this had not affected RHEL and Fedora stable branches. While no Fedora stable builds were affected, Fedora 40 beta nightly builds, the leading edge of Linux innovation, were affected.

The collaboration kept moving swiftly throughout Thursday afternoon (in US time zones), with our InfoSec team delivering the first YARA rules, ProdSec evaluating the Vulnerability Management aspects (which version affects what), and the ProdSec Supply Chain team verifying if our productization pipeline was sound and not using affected versions.

Early Thursday morning, Red Hat and IBM executives were briefed about this incident and given a complete picture of what was known so far.

****Multi-vendor collaboration****

At this stage, distro-lists at Openwall were beginning to see some traffic, but due to the limited reach of this community, we engaged the US Cybersecurity and Infrastructure Security Agency (CISA) to start a collaborative Vulnerability Information and Coordination Environment (VINCE) ticket. This enabled other vendors not participating in distro-lists, as well as the vulnerability discoverer by Andres Freund, to be engaged continuously.

Collaboration continued to flow with the ticket creation, moving around the clock through 48 participating entities: BSDs, proprietary software vendors, national Security Response teams and the heavyweights of the hardware and software landscape were working together to advance and level the field on the findings, investigation progress, other products’ affectedness and more, all done at speed to make the issue public as soon as possible.

****To CVE or not to CVE****

Historically, malware has been categorically and intentionally omitted from the CVE ecosystem. They are typically beyond the control of the vendors who provide software. However, in this case it was embedded into the vendor code, there was a vulnerable condition and relying on antivirus software to detect the existence of this malicious software can leave room for detection failure.

During the discussion with other CVE Numbering Authorities (CNAs), we erred on the side of caution. This is a condition that:

• Has to be looked for and detected
• Compromises confidentiality, integrity, and availability
• Is described by an existing CWE type
• Is a type of compromise that has a precedent of assigned CVEs
• Affects a very specific name-version-release (NVR)
• Is a high profile issue that needs the CVE ID as an industry-wide common identifier

That said, as a CNA, we decided to assign a CVE for this vulnerability: CVE-2024-3094.

****Good Friday****

On Friday, all the ducks (and there were a lot of them) were lined up and while this probably ruined a fair number of holidays, the participants in the VINCE ticket, including the finder, agreed to bring this to the public as soon as possible. As soon as Andres’ post went live in OSS-Security, CISA alerted the general public and Red Hat published a post as well. We had our Communications team fully engaged, informed and ready for the inquiries. Minutes after disclosure in OSS-Security, we started receiving inquiries from major news outlets asking for more information.

The impacted distros immediately pulled back the affected libraries and prepared statements to their user bases.

It was good (no pun intended) that in the end, this attack only affected a very limited set of worldwide systems in a small number of leading-edge distributions, including Fedora, and there was no evidence of further tampering or manipulation in affected systems.

To err on the side of safety, all Red Hat assets that were running the affected Fedora versions were deemed as tainted and were required to be reinstalled or redone.

Although this event had the potential to really hurt the open source community, at the end of the day, it actually displayed its strength. Multiple people, organizations and contributors worked together to stop a very real threat. The imminent incident has been stopped and remediated, but the work is not done. Many lessons have been learned around the world from this event, and I think it is fair to say that the open source community will implement improvements to minimize future risks with our distributions.