Black Hat Asia: Firmware Supply-Chain Woes Plague Device Security

BLACK HAT ASIA 2022 — When it comes to developing the firmware that powers computing devices, the ecosystem consists of complex supply chains that have multiple contributors. For any given device, firmware could be made up of a hodgepodge of components from different sources. And that means that when it’s time to address security vulnerabilities, it’s far from a straightforward process to get a patch out to the public.

During a panel-discussion session at Black Hat Asia on Thursday, entitled "The Firmware Supply-Chain Security Is Broken: Can We Fix It?", Kai Michaelis, co-founder and CTO at Immune GmbH, outlined what he called the overgrown supply-chain “tree,” out of which grows onerous code reviews, and lengthy patching processes when a bug is found.

In fact, six to nine months for patches to roll out is the average, according to the panelists — with two years being not uncommon. And that means the supply chain represents a wide attack surface that’s ripe for compromise, they warned. Given that vulnerable firmware threatens safety of the operating system and any applications, the potential for cyberattackers to find exploitable vulnerabilities is a serious concern.

A Thorny Tree of Supply-Chain Complexity

The final firmware that vendors incorporate into their hardware is a multisourced affair, explained Michaelis. Stakeholders can include various component vendors, a few open source repositories, reference implementations, original design manufacturers, independent BIOS vendors, and finally, the original equipment manufacturers (OEMs) that create and sell the final product to channel partners and end users.

Further complicating matters is the fact that subsystem vendors might be sitting in the middle of the code tree, itself combining elements from multiple component manufacturers into a single offering.

The unfortunate end result is that when a vulnerability is reported, OEMs often have multiple “branches” from which patches and updates flow — and they usually have no visibility to each other.

“It’s a tree of suppliers and updates with little coordination between them, and the OEM has to ingest all of it,” Michaelis said. “For vendors, packaging updates is a fairly manual process, and then consumers need to actually install those updates. In all, the patching process as it stands can be measured in months to years.”

One of the main issues that Michaelis flagged is the fact that when bugs are found, they may be benign in and of themselves. However, when combined with additional vulns in other parts of the firmware, the flaws become weaponizable and could allow attacks on value-added reseller (VAR) partners — and from there, end users.

“Convincing a vendor to patch what it believes is a harmless flaw is not easy,” he said. “And even if there is a patch, it takes so long for it to get downstream that an attacker could easily find another vulnerability to combine with it in the meantime. So this is the problem: Bugs exist in isolation because vendors don’t talk to each other, and bugs have a long shelf life.”

There are at least three other aspects that make matters even worse: One, end-of-life (EoL) devices often don’t get updates; two, each vendor follows its own patch cycle; and three, sometimes vendors offer silent updates without issuing an advisory, which can discourage OEMs from incorporating patches.

Repeating the Same Mistakes

Alex Matrosov, founder and CEO at Binarly, explained during the panel that like in the software supply chain, firmware bugs can also be spread and re-imported even after they’ve been patched, resulting in what he called “repeatable failures.”

For instance, a bug recently disclosed in one of the components in the Intel M15 laptop kit (CVE-2022-27493) is a classic out-of-bounds write flaw stemming from system-management mode (SMM) memory corruption — but not as what it seems.

“It’s actually a 2019 bug found in the AMI codebase that we’ve now discovered in 2022 firmware,” Matrosov explained. “This vulnerability was fixed, but the fixed version was not included by the device vendor. It’s a very vulnerable component and has been known for years as a suitable attack vector, and it should be removed.”

In another example, vulnerable code in an EDK open source library called SecurityPkg was removed in EDK II in 2018. However, somehow it found its way into 2022 firmware affecting several OEMs, via another library. “The risk was exponentially compiled,” Matrosov said.

Best Principles for Pruning Back the Patching Misery

So, what’s to be done? According to the panel, it will take a profound shift in strategy and thinking to reliably shore up firmware security. However, a good place to start is an aspirational list of first principles.

The panelists advocated, for instance, that OEMs and members of the security community as a whole make a concerted effort to educate component vendors and other supply-chain elements about security and convince them that updates are a necessity, even for EoL devices — and that further, if they don’t issue a CVE, it becomes more difficult to communicate the urgency to patch and the bugs become difficult to track.

OEMs also should put in place efforts to increase risk transparency, according to the panel. This can be done by facilitating greater communication between vendors and creating a centralized repository of information about patches and bugs.

“Fixing the supply chain is a team sport,” Matrosov said, noting that working with computer emergency response teams (CERTs) is a good goal.

“We really need an independent body to help coordinate patches when they affect multiple vendors, and to facilitate simultaneous patching. If one vendor patches and another doesn’t, it creates a dangerous zero-day situation for a subset of the devices,” he added.

Private security community collaboration will also be key, the panelists said. For instance, the Linux Foundation has launched a website called LVFS, which is a vendor firmware service that allows OEMs to upload firmware updates to be distributed to Linux users at zero cost. So far, about 150 vendors are participating, including Dell, HP, Intel, and Lenovo.

“There are about 1,000 different devices supported, and we’ve shipped more than 51 million updates since we started the project,” said panelist Richard Hughes, principal engineer at Red Hat. “Also, we can take the firmware and decompress it into shards. A shard might be an EFI, binary, Intel microcodes, AMD PSP image, etc. So, all of those vendors uploading all those updates gives us a huge amount of data.”

From there, the system can show users, say, the newest available Intel microcode for all of the different models in the system — and can push updates automatically.

There’s plenty to be done, but Hughes struck an optimistic note.

“My personal conclusion is that by working together with CERTs and security companies, we can improve the immune system even further, speeding up the process of shipping fixes to end users and making sure that security issues patched by all vendors,” Hughes said. “These are really hard problems that have plagued the entire industry for 20 years. Only now do we have all the infrastructure and the data to make things better.”