Understanding the Confidential Containers Attestation Flow

Introduction to attestation

Attestation is a confidential computing keystone. With attestation, workload owners can fully assert the trustworthiness of the hardware and software environment their workload is running in, regardless of the security posture of the underlying infrastructure provider.

This article describes the hardware-based attestation flows and processes that the Confidential Containers project is built upon. With hardware-based attestation, a confidential computing processor generates cryptographic evidence for a workload-running environment. Provided that the workload owner trusts that piece of hardware, they can then remotely verify that evidence and decide if the workload’s execution environment is trustworthy or not. If it is, the owner can then provision it with a set of secrets ( e.g. container image encryption keys), effectively permitting it to run the workload.

The Background-Check” model

The Confidential Containers (CoCo) project follows the IETF Remote Attestation Procedures (RATS) “background check” model as shown in the diagram below:

Source: https://ietf-rats-wg.github.io/architecture/draft-ietf-rats-architecture.html

The following are the key roles in the attestation process:

Attester

The Attester is the entity that aims at being attested by an external, trusted service called the Relying Party. This is in order to be provisioned back with a secret (e.g. an encryption key from Key Management Service) that’s only released to trustworthy entities. In practice, an Attester is usually a piece of software running on a device, in a virtual machine or, more generally, in a cloud workload environment.

Attesters inform the Relying Party of the state of the system they’re running on by sending an Evidence. The Evidence informs the Relying Party about the state of the system, the Trusted Computing Base (the part of hardware, firmware and software that is trusted) and other aspects of the system. It comprises a set of claims, later asserted or denied by the Verifier. The Evidence is cryptographically signed with a key (e.g., from the silicon vendor) used during the verification process.

The Confidential Containers Attester is implemented via the attestation agent project.

Relying Party (Key Broker Service)

The Relying Party is usually made of a set of trusted services that run outside of the trusted execution environment, either on the same cluster (for local Attestation) or on a different, physically separated infrastructure (for remote Attestation). Its main role is to provision the Attester with some secret (an encryption key, a token, a passphrase, etc.) after assessing the Attester’s trustworthiness. This is achieved by verifying cryptographically signed Evidence of the Attester’s security posture (generally including its Trusted Computing Base (TCB) measurements, configuration, software versions, etc.).

More formally, the Relying Party’s primary functions are:

• Receive Evidences from Attesters (confidential virtual machine (VM) or container) via a challenge-response protocol
• Relay the Evidences to an Attestation Service (including a Verifier) for verification. The Attestation Service returns Attestation Results back to the relying party
• Apply appraisal policies to the Attestation Results in order to assess the trustworthiness of the Attester. An appraisal policy is defined as a set of constraints to apply the Attestation Results or to an Evidence. Examples of such constraints include checking for equality with Reference Values or membership in a set of Reference Values
• Upon successfully verifying the Attester’s trustworthiness, interact with the Key Management Service to retrieve secrets and send them to the Attester

The Confidential Containers project defines a Relying Party protocol and API through the Key Broker Service (KBS) project. The reference-kbs is a reference implementation for that protocol.

Verifier (Attestation service)

The Attestation service verifies the Evidence based on configured policy and reference values. Think of reference values as “good” and “trusted” values, which are known beforehand and used to verify the Evidence sent by the Attester. These reference values are typically generated when building the system software and firmware layers through a coupled CI/CD pipeline.

This is implemented by the Attestation Service project under the Confidential Containers umbrella.

Key Management Service

A security-focused service for storing, managing and backing up cryptographic keys used by applications and users. Usually, the secrets that are sent to the Attester on successful attestation are stored in the Key Management Service (KMS).

Confidential Containers Attestation

Although the Confidential Containers software architecture follows the above-described attestation general principles, it does so in a particular software context: Kubernetes and the cloud-native ecosystem.

The following sections will first go through the threat model to fully understand the specifics and rationale behind the Confidential Containers attestation flow. Highlighting the security threats the project aims to mitigate will make its architecture and implementation easier to grasp and understand.

Threat model

The Confidential Containers attestation flow aims at protecting Kubernetes-specific assets from threat agents.

Agents

In a typical Kubernetes context, the infrastructure provider (such as a public cloud provider) is not considered a threat agent. It is a trusted actor of a Kubernetes deployment.

In a confidential computing context, that assumption no longer applies and the infrastructure provider is a potential threat agent. Confidential Computing in general, and Confidential Containers in particular, try to protect Kubernetes workload owners from the infrastructure provider. Any software component that belongs to the infrastructure (e.g. the Kubernetes control plane) is untrusted.

Assets

The main asset the Confidential Containers project protects is the Kubernetes workload itself, i.e.

• The Kubernetes pod container images
• The data generated and potentially stored by the pod at runtime, running from its container images

Threat vectors

On top of the generic Confidential Computing threat vectors, the CoCo threat model defines a few Kubernetes-specific ways the above-described assets can be compromised:

1. The infrastructure provider tampers with or reads the Kubernetes pod container images when they’re either pulled or stored
2. The infrastructure provider tampers with or reads the Kubernetes pod memory while in use
3. The infrastructure provider tampers with or reads data stored (ephemerally or not) by the Kubernetes pod

Mitigations

As CoCo aims at protecting the Kubernetes workload from the infrastructure owner, its threat model imposes a fundamental architectural shift compared to traditional Kubernetes deployments. Relating to the previous threat vectors, let’s discuss how they are addressed.

With Confidential Containers, the pod container images are no longer controlled by the infrastructure owner and must be signed or encrypted. In other words, the infrastructure owner must not pull, store and manage the container images. The workload itself must be able to pull, decrypt or verify, and store (ephemerally or not) container images. Having the encrypted or signed images stored in confidential memory or storage guarantees that the infrastructure owner can never access them, including while they are being pulled from container registries through the infrastructure provider networking fabric. This addresses threat vector #1.

Additionally, the CoCo threat model requires that the workload runs exclusively in confidential memory, which provides a hardware guarantee against the infrastructure provider tampering with Kubernetes pods’ memory. This addresses threat vector #2.

Finally, Confidential Containers implements trusted storage, where Kubernetes pod’s volumes must be encrypted and integrity protected. Confidential Containers will not mount or create Kubernetes volumes that are not protected with attestation injected secrets, effectively preventing infrastructure providers from tampering with workload owners’ stored data. This addresses threat vector #3.

Architecture and flow

The CoCo attestation architecture’s main objective is to allow for implementing the threat mitigations described in the previous section.

The key requirement for those mitigations is the ability for the CoCo guest to pull, decrypt, unpack and store the container images that comprise the Kubernetes pod. From this requirement, we can derive a set of mandatory attestation attributes for CoCo:

• The CoCo guest image is trusted, measured and must be attested
• The trusted guest image pulls and stores encrypted or signed container images
• The container images are not measured nor attested (see the next section for details)

Given that the main asset CoCo is protecting from the infrastructure owner is the Kubernetes pod and its container images, why are they not measured and attested? There are two main issues with measuring the container images:

1. Measuring a multi-gigabyte container image can be CPU-intensive and significantly increase the pod startup time
2. Adding all container image measurements to the Attestation Evidence means they must effectively be part of the CoCo guest stack loaded at pod startup time. This model’s lack of flexibility (One virtual machine image per Pod) makes it a cloud-native anti-pattern

This is where the power of hardware-based, remote attestation comes into play, as it allows for building a much more dynamic model.

Although container images are not measured, they are encrypted, or at least signed, by an entity that the workload owner trusts. That entity could be either the workload owner itself or an entity that the owner trusts with safely building, signing and encrypting its container images.

The CoCo guest will then use remote attestation to decrypt the container images. It will initiate the remote attestation process before trying to pull any of the container images, asking for keys bound to those images. Upon attestation success, the Relying Party will provision the CoCo guest with those keys, allowing the former to pull and either decrypt or verify the container images.

Because the CoCo attestation architecture requires the container images to be signed or encrypted, the workload owner does not have to trust the infrastructure networking fabric or the container image registries from which the container images are downloaded.

Finally, and most importantly, as the workload owner can attest to the CoCo guest’s trustworthiness before releasing the container images encryption keys, it knows that the infrastructure provider cannot see or tamper with its workload.

In summary, the CoCo attestation architecture can be defined through a few key points:

• The guest image and configuration are the only measured and attested parts of the CoCo stack. Container images are neither measured nor attested
• At runtime, the guest will use remote attestation to get a set of decryption keys for the container images from a Relying Party
• By letting an attested CoCo guest decrypt and verify the Kubernetes pod container images with keys received from a Relying Party, the workload owner no longer has to trust the infrastructure provider
• Encrypted or signed images are protected from the infrastructure provider or the container image registries trying to read or tamper with the workload

The following section will describe how the Confidential Containers project implements this attestation architecture.

Implementation

The Confidential Containers project integrates several open-source software components into a measured software stack that constitutes the Confidential Containers guest. This guest stack runs in a hardware-virtualized environment controlled and managed by the Kata Containers container runtime. This runtime creates and assigns Kubernetes pods to their dedicated virtual machine where the pod workload will eventually run. The CoCo guest is defined as the set of trusted components that boot and run the Kata Containers virtual machine, from the guest firmware to the attestation and Kata agents. It will pull and decrypt container images from the virtual machine itself, unlike non-confidential Kata Containers guests who receive the pod images from the untrusted host.

The Confidential Computing hardware (A CPU or a System-on-Chip) measures all the CoCo guest components and state when it loads them, and adds those measurements to any Attestation Evidence that it may generate. It is important to note that:

1. As integral parts of the workload’s trusted domain, all the CoCo guest components are verified by the Kubernetes workload owner through the attestation process.
2. The container images themselves are not measured and are not included in the Attestation Evidence.

Some of the Confidential Containers components are developed and maintained through the project’s github organization, while others are external components. The components that implement the Confidential Containers attestation flows are:

• The attestation-agent is the main attestation component for the Confidential Containers project. It is responsible for initiating and running the attestation process in order to eventually receive a container image encryption key from a relying party, i.e. from the workload owner. The attestation-agent follows a modular architecture that allows for supporting multiple attestation protocols. Each attestation module implements the Key Broker Client (KBC) interface to support a different Key Broker Service (KBS) protocol
• The image-rs container image management library is used by the Kata Containers agent to pull, unpack and verify or decrypt container images within the CoCo guest. To decrypt or verify a container image, the image-rs library relies on the ocicrypt-rs library
• The ocicrypt-rs library implements the OCI container image encryption specification and is used by image-rs to decrypt and verify a pulled container image. In order to do so, ocicrypt-rs use the above-described attestation agent as a key provider. The latter runs the attestation flow to eventually receive a container image decryption key that will be consumed by ocicrypt-rs

The following diagram describes how the attestation components fit into the overall CoCo architecture:

Source: https://github.com/confidential-containers/documentation/blob/main/architecture.md

The red connectors in this diagram describe the CoCo attestation flow:

1. The Confidential Containers runtime (i.e. Kata Containers) on the host initializes and starts the virtual machine
2. The enclave software stack (i.e. the CoCo guest stack) boots the virtual machine
3. For each container composing the pod, the enclave agent (i.e. the Kata Containers agent) pulls the container image manifest by calling into the image-rs API
4. Before pulling the container images, image-rs parses the image manifest to understand which image layers are encrypted
5. For each layer that is to be decrypted, image-rs calls into ocicrypt-rs in order to get the associated decryption key
6. ocicrypt-rs calls into the attestation-agent API in order to get the decryption key for that layer. The attestation agent is ocicrypt-rs’s key provider
7. The attestation agent sends a request for the decryption key to the workload’s owner, a.k.a the Relying Party (Key Broker Service). This is the initial step of the remote attestation procedure
8. The Relying Party replies by sending an Attestation Challenge back to the attestation agent
9. The attestation agent replies to that Challenge with an Attestation Evidence for the CoCo guest TCB. The evidence includes all of the CoCo guest components and states, and is signed by the Confidential Computing hardware. This is step 1 on the above diagram (red arrows)
10. The Relying Party forwards the Evidence to the Attestation Service, which performs the Evidence verification. This is step 2 on the diagram
11. The Attestation Service sends the Attestation results back to the Relying Party. If those results show a successfully verified Attestation Evidence, the Relying Party is authorized to get the decryption keys from the Key Management Service (step 3)
12. The Relying Party Response to the attestation agent challenge includes the container image decryption keys. The keys are wrapped, i.e. they’re encrypted with a confidential computing backed public key (step 4)
13. Now that it has all the needed decryption keys, the attestation agent pulls the container images. This is step 5 on the above diagram
14. After pulling all images, the attestation agent calls into the image-rs and ocicrypt-rs APIs to decrypt and then unpack the container images in the enclave (step 6)
15. The attestation agent starts and runs the Kubernetes pod workload, from the container images that it pulled, decrypted and unpacked through the previous steps

**Putting it all together **

Let’s now take a working example of using an encrypted container image and see how the different components previously described fit together.

Note that the process of creating an encrypted container image is outside of the CoCo workflow today. The CoCo stack is focused on deploying and running the encrypted image. Please refer to the following external blog for details on encrypted container images.

When you create a Kubernetes POD with an encrypted container image, the components described under the Decryption box (right pink block) get used as following:

1. The Kata containers runtime spins up a confidential VM (the overall block)
2. The image-rs component of kata-agent uses the Attestation Agent to start the attestation process
3. This will kickstart the attestation process by sending the evidence to the Key Broker Service (KBS) - the relying party
4. The KBS uses the verifier service to verify the evidence and on success gets the required secrets from the Key Management Service (KMS) and provides it to the Attestation Agent
5. The image-rs component then uses the secret to decrypt the image and start the container

Summary

In this blog, we discussed Attestation, an essential aspect of Confidential Computing, in the context of the CNCF Confidential Containers project.

We introduced the confidential containers threat model and looked at the Attestation workflow, including the implementation details.

In future blogs, we will touch upon Attestation examples from different confidential containers environments. We will also dive into the detailed implementation of the different components mentioned here.