Dangerous New Attack Technique Compromising VMware ESXi Hypervisors

VMware issued urgent new mitigation measures and guidance on Sept. 29 for customers of its vSphere virtualization technology after Mandiant reported detecting a China-based threat actor using a troubling new technique to install multiple persistent backdoors on ESXi hypervisors.

The technique that Mandiant observed involves the threat actor — tracked as UNC3886 — using malicious vSphere Installation Bundles (VIBs) to sneak their malware onto target systems. To do so, the attackers required admin-level privileges to the ESXi hypervisor. But there was no evidence that they needed to exploit any vulnerability in VMware’s products to deploy the malware, Mandiant said.

Wide Range of Malicious Capabilities

The backdoors, which Mandiant has dubbed VIRTUALPITA and VIRTUALPIE, enable the attackers to carry out a range of malicious activities. This includes maintaining persistent admin access to the ESXi hypervisor; sending malicious commands to the guest VM via the hypervisor; transferring files between the ESXi hypervisor and guest machines; tampering with logging services; and executing arbitrary commands between VM guests on the same hypervisor.

“Using the malware ecosystem, it is possible for an attacker to remotely access a hypervisor and send arbitrary commands that will be executed on a guest virtual machine,” says Alex Marvi, a security consultant at Mandiant. “The backdoors Mandiant observed, VIRTUALPITA and VIRTUALPIE, allow attackers interactive access to the hypervisors themselves. They allow attackers to pass the commands from host to guest.”

Marvi says Mandiant observed a separate Python script specifying which commands to run and which guest machine to run them on.

Mandiant said it was aware of fewer than 10 organizations where the threat actors had managed to compromise ESXi hypervisors in this manner. But expect more incidents to surface, the security vendor warned in its report: “While we noted the technique used by UNC3886 requires a deeper level of understanding of the ESXi operating system and VMware’s virtualization platform, we anticipate a variety of other threat actors will use the information outlined in this research to begin building out similar capabilities.”

VMware describes a VIB as a “collection of files packaged into a single archive to facilitate distribution.” They are designed to help administrators manage virtual systems, distribute custom binaries and updates across the environment, and create startup tasks and custom firewall rules on ESXi system restart.

Tricky New Tactic

VMware has designated four so-called acceptance levels for VIBs: VMwareCertified VIBs that are VMware created, tested, and signed; VMwareAccepted VIBs that are created and signed by approved VMware partners; PartnerSupported VIBs from trusted VMware partners; and CommunitySupported VIBs created by individuals or partners outside the VMware partner program. CommunitySupported VIBs are not VMware- or partner-tested or supported.

When an ESXi image is created, it is assigned one of these acceptance levels, Mandiant said. “Any VIBs added to the image must be at the same acceptance level or higher,” the security vendor said. “This helps ensure that non-supported VIBs don’t get mixed in with supported VIBs when creating and maintaining ESXi images.”

VMware’s default minimum acceptance level for a VIB is PartnerSupported. But administrators can change the level manually and force a profile to ignore minimum acceptance level requirements when installing a VIB, Mandiant said.

In the incidents that Mandiant observed, the attackers appear to have used this fact to their advantage by first creating a CommunitySupport-level VIB and then modifying its descriptor file to make it appear that the VIB was PartnerSupported. They then used a so-called force flag parameter associated with VIB use to install the malicious VIB on the target ESXi hypervisors. Marvi pointed Dark Reading to VMware when asked whether the force parameter should be considered a weakness considering that it gives administrators a way to override minimum VIB acceptance requirements.

Operation Security Lapse?

A VMware spokeswoman denied the issue was a weakness. The company recommends Secure Boot because it disables this force command, she says. “The attacker had to have full access to ESXi to run the force command, and a second layer of security in Secure Boot is necessary to disable this command,” she says.

She also notes that mechanisms are available that would allow organizations to identify when a VIB might have been tampered with. In a blog post that VMWare published at the same time as Mandiant’s report, VMware identified the attacks as likely the result of operational security weaknesses on the part of the victim organizations. The company outlined specific ways organizations can configure their environments to protect against VIB misuse and other threats.

VMware recommends that organizations implement Secure Boot, Trusted Platform Modules, and Host Attestation to validate software drivers and other components. “When Secure Boot is enabled the use of the ‘CommunitySupported’ acceptance level will be blocked, preventing attackers from installing unsigned and improperly signed VIBs (even with the --force parameter as noted in the report),” VMware said.

The company also said organizations should implement robust patching and life-cycle management practices and use technologies such as its VMware Carbon Black Endpoint and VMware NSX suite to harden workloads.

Mandiant also published a separate second blog post on Sept. 29 that detailed how organizations can detect threats like the one they observed and how to harden their ESXi environments against them. Among the defenses are network isolation, strong identity and access management, and proper services management practices.

Mike Parkin, senior technical engineer at Vulcan Cyber, says the attack demonstrates a very interesting technique for attackers to retain persistence and expand their presence in a targeted environment. “It looks more like something a well-resourced state- or state-sponsored threat would use, versus what a common criminal APT group would deploy,” he says.

Parkin says VMware technologies can be very robust and resilient when deployed using the company’s recommended configurations and industry best practices. “However, things become much more challenging when the threat actor is logging in with administrative credentials. As an attacker, if you can get root you have the keys to the kingdom, so to speak.”