Confidential computing platform-specific details

Confidential Computing is a set of technologies designed to protect data in use (for example, it provides memory encryption). This article is fifth in a six-part series (see the previous article), about various Confidential Computing usage models, and the requirements to get the expected security and trust benefits.

In this article, I explore the many available Confidential Computing platforms, and discuss how they differ in implementation, and specifically in how to perform attestation:

• AMD Secure Encrypted Virtualization (SEV) in its three generations (SEV, SEV-ES and SEV-SNP)
• Intel Trust Domain Extensions (TDX)
• IBM Z Secure Execution (SE)
• OpenPOWER Protected Execution Facility (PEF)
• ARM Confidential Compute Architecture (CCA)

At the moment, Qemu officially supports AMD, Power and IBM Z, and Intel maintains branches for TDX. Confidential Containers has some support for SEV and SEV-ES (SEV-SNP is in development), SE on IBM Z, and Intel TDX. This is changing rapidly.

In the cloud, providers are in the process of deploying SEV-SNP support. Separate Linux kernel patches are required for host and guest support. Generally, the status of guest support patches is a bit ahead of host support. The various use cases (confidential virtual machines, workloads, containers, and clusters) are also at different stages in their support of the technologies.

AMD Secure Encrypted Virtualization (SEV)

AMD provides Confidential Computing through a technology called secure encrypted virtualization (SEV). This technology builds on top of virtualization. It relies on a separate security processor running an independent firmware from the primary x86 cores.

Initial iterations of the AMD SEV technology only allow pre-attestation (see chapter 6 in SEV encrypted virtualization API), in which a measurement of memory content is done before a virtual machine is even started. The attestation process delivers a launch secret. In the first phase, the hypervisor loads the initial boot state in memory, then asks the firmware to encrypt and measure it. The measurement is sent to the attestation service, which responds with a launch secret. Only then does the hypervisor run the guest. The guest can then use that launch secret in a guest-specific way (for example, to access an encrypted root disk).

This only allows a crude form of system-facing attestation, where the attestation process itself is not under control of the guest, but is executed by the host. Additionally, several serious vulnerabilities have been found in this scheme ( see undeSErVed trust, Insecure until Proven Updated).

AMD SEV-Encrypted State (SEV-ES)

While the original SEV only encrypts the contents of guest memory, SEV-ES also encrypts the contents of the guest CPU registers. This stops leakage of in-flight values and, more importantly, stops a malicious hypervisor from manipulating registers to read out encrypted data. It does not, however, significantly change the attestation model relative to the earlier generation.

AMD SEV-Secure Nested Pages (SEV-SNP)

SEV-SNP is the third generation of SEV, introduced in 2021. The EPYC CPU adds defense against malicious host manipulation of page mapping, but also features two changes that impact attestation:

1. SNP provides a mechanism to obtain the attestation quote inside the guest once the guest is running (earlier forms of SEV had to perform attestation from the hypervisor). This makes it much easier for the guest to pass a quote to the attestation server. Of course, the request is encrypted before it’s sent to the firmware, so that the hypervisor has no access to it.

2. SNP includes an extra set of privilege level separation in the guest called virtual machine privilege level (VMPL). This allows the secure implementation of a virtual TPM, protected by the same SEV machinery protecting the rest of the guest. A new piece of firmware, the Secure Virtual Machine Service Module (SVSM) runs with highest privilege (VMPL0) and is planned to incorporate the vTPM implementation. Support firmware and software is discussed in more detail in the next article of this series.

The attestation mechanism, in the case of SEV-SNP, is described by AMD using the following schema:

Intel Software Guard Extensions (SGX)

Intel SGX provides process-level isolation, allowing the creation of secure enclaves. This makes it unique among the trusted execution environments discussed here. There is some support for this technology in the Confidential Containers project, under the name enclave-cc. This approach supports both local and remote attestation.

Because it is process-based, that technology requires some additional user-space trickery to be able to run (or emulate) multi-process workloads, such as the ability to fork a new process. This is marked as LibOS on the diagram.

On the other hand, it draws the boundary of the trusted computing base much closer to the container you want to run, because there is no independent kernel. This reduces the attack surface and resource requirements significantly.

The SGX technology was deprecated in 2021, except on the Xeon line of processors. However, it plays a key role in TDX attestation, to provide a so-called quoting enclave.

Intel Trust Domain Extensions (TDX)

Intel Trust Domain Extensions (TDX) provide features similar to AMD SEV-SNP. It offers memory and state encryption and integrity to isolate trust domains (TD, confidential virtual machines) from each other, as well as from the host operating system and hypervisor. TDX also addresses state and interrupt integrity much like AMD SEV-SNP, and has methods to ensure that vulnerabilities in the TDX firmware itself can be securely fixed, and that insecure firmware can be remotely detected.

A major design difference compared to AMD’s approach is that this is implemented using a new processor mode called secure arbitration mode (SEAM), rather than on a separate security processor. The memory used by SEAM is in a reserved protected range, defined by dedicated registers in the processor. The main x86 processor enters this special SEAM mode using new dedicated SEAMCALL instructions. In short, the design is based on protected firmware rather than a dedicated security processor.

The calls are intended to invoke services provided by a special firmware authenticated code module (ACM) provided by Intel. The owners of a TDX system, for example a cloud provider, can define which versions of this module are acceptable. The integrity of this digitally signed module is verified using a TPM by Intel Trusted Execution Technology (TXT).

Because there is no security processor, the TDX architecture relies on a quoting enclave, an SGX enclave running code provided by Intel that takes advantage of the special instruction SEAMREPORT to generate a quote for a given trusted domain.

For the same reasons, providing a virtual TPM on a TDX platform presumably uses an enclave or a trusted domain. A design for such a virtual TPM based on a trusted domain storing the data for the various virtual TPMs has been presented by Intel.

Similar to AMD-SEV, having a virtual TPM simplifies the deployment of existing workloads, or in other words, deployments that have not been enlightened specifically for TDX.

IBM Z Secure Execution (SE)

The IBM Secure Execution capabilities is available for the IBM z15 and z16 mainframes. It is a somewhat different design compared to what exists on x86, introducing a new level of system management in the firmware called the ultravisor that sits below the hypervisor.

Hardware encryption of memory and remote attestation were introduced with the z16. On the z15, the ultravisor blocks access from the host, but physical memory is not encrypted. This design entrusts firmware with control and execution of workloads, which are encrypted and stored with a special SE header. This makes it possible to restrict execution to a particular host or set of trusted hosts.

The attestation approach is also different. IBM’s point of view is that “IBM Secure Execution does not require external attestation to prove that a guest is secure. If the image contains a unique secret, a successful login implicitly attests an SE guest image.”

However, they still added remote attestation. This provides more flexibility when proving a larger set of properties, allowing multiple instances of the same image with instance-specific secrets, presenting a proof to a third party, or proving some property of the execution environment.

Power Protected Execution Facility (PEF)

The Protected Execution Facility (PEF) on OpenPOWER is reminiscent of the Secure Execution on IBM z. It’s based on an additional ultravisor protection level, except this firmware layer is open source. This platform uses the terminology secure virtual machines (SVM) rather than "confidential". In order to run properly on the ultravisor, the hypervisor needs to be paravirtualized (for example, use ultravisor services for some operations using a new ESM instruction).

The boot process starts the VM normally, and then performs an ultravisor call to copy the VM address space into secure memory. It then searches for a valid authorization key for a specific machine. A bit like on IBM z, the base attestation model is to prove that you are running on one of the allowed machines. However, the SVM itself can perform a remote attestation, notably in order to implement revocation. IBM presentations highlight that the user, the owner, or the host of an SVN all may be given the opportunity to revoke an SVM.

ARM Confidential Compute Architecture (CCA)

For the ARM platform, the confidential computing technology is called confidential compute architecture (CCA). As on Power, it is based on the introduction of a new exception level (EL3), sitting below the hypervisor. This enables a monitor mode. The architecture builds on ARM v8 TrustZone, which was used to split between a “Secure world” (to run trusted applications and operating systems) and a “Normal” world (for standard applications and operating systems). CCA introduces a new concept of realm, which is a partitioning of the address space in order to isolate realms from one another, controlled by a new data structure called the granule protection table (GPT), and managed by a new firmware component called the realm management monitor (RMM). The initial root of trust in the system is called the runtime security subsystem (RSS).

Attestation on CCA is managed primarily by new firmware that takes advantage of the new facilities, and can build the required attestation reports.

This architecture is still under development.

Conclusion

Available architectures take different approaches to the same problem, with different emphasis being placed on hardware, software and firmware responsibilities. All the solutions provide roughly the same confidentiality guarantees, usually with the ability to encrypt memory, measure it using cryptographic algorithms for attestation purposes. Each can offer some provable integrity guarantees about the initial state of a confidential virtual machine, including the fact that it’s running in a trusted execution environment. In the next article, I look at some interesting support technologies where active open source development is happening.