Introducing confidential containers on bare metal

Confidential Containers (CoCo) are containers deployed within an isolated hardware enclave protecting data and code (data in use) from privileged users such as cloud administrators. Red Hat OpenShift confidential containers are available from OpenShift sandboxed containers 1.7.0 as a tech-preview on Azure cloud and as a tech-preview on Azure Red Hat OpenShift.

In this article we introduce confidential containers on bare metal which is now available as a preview using Assisted Installer for OpenShift. We cover a number of use cases for CoCo bare metal, explain how it works with different trusted execution environment (TEE) technologies (Intel TDX and AMD SEV-SNP) and third party attestation services such as Intel Trust Authority (ITA). We provide a high level overview of how this solution is deployed today, using an on-premises environment as an example. Additionally we briefly discuss the future roadmap and known issues in this preview release.

We finish up with a demonstration of deploying CoCo bare metal and using it with Red Hat OpenShift AI.

**OpenShift confidential containers for bare metal **

The OpenShift confidential containers solution is based on deploying 2 operators:

• OpenShift sandboxed containers operator—based on the Kata Containers open source project running pods inside virtual machines (VMs) for isolation workloads. This operator has been enhanced to also support the CNCF Confidential Containers (CoCo) open source project when deployed in environments that support TEE infrastructure such as Intel TDX and AMD SEV-SNP. For simplicity we refer to this operator in diagrams as “OpenShift confidential containers.”
• Confidential compute attestation operator—based on the Trustee project (also part of the CNCF CoCo open source project) providing remote attestation capability. It’s responsible for performing the attestation operations and delivering secrets after successful attestation. A key point for this operator is to be deployed in a trusted environment

For additional details on the building blocks each operator consists of, their interaction and the key problems they solve (workload secret, signed container image, encrypted container image), we recommend reading Exploring the OpenShift confidential containers solution and Use cases and ecosystem for OpenShift confidential containers.

In this article our starting point is the following diagram showing the overall OpenShift confidential containers deployed for secrets retrieval by the workload use case:

CoCo integrates TEE infrastructure with the cloud-native world. A TEE is at the heart of a confidential computing solution. TEEs are isolated environments with enhanced security (e.g. runtime memory encryption, integrity protection), provided by confidential computing-capable hardware. A special virtual machine (VM) called a confidential virtual machine (CVM) that executes inside the TEE is the foundation for OpenShift CoCo solution.

For CoCo bare metal, our focus is around TEEs provided by Intel TDX and AMD SEV-SNP.

The primary driver for CoCo bare metal solution has been for on-prem environments.

CoCo bare metal business use cases for on-prem environments

Whether to use CoCo on-prem or CoCo public cloud depends on the area of trust to be established:

• Public cloud is useful when a zone of trust between two or more partners in an untrusted area is required
• On-prem is useful when increased isolation from other services is required— deploying a workload which must be run inside of a company’s trusted environment in order to give partners a safe haven. In this case, the company deploying the workload provides the servers and the host systems. Therefore, this environment usually consists of bare metal installations. The purpose here is to provide an extra level of security to either partners or (internal/external) customers within an on-prem datacenter

Let’s look at some actual CoCo on-prem business use cases.

IP protection / IP integrity

Let’s say a supplier is interested in providing a service to a customer. This service includes running workloads in the customer’s environment, and these workloads include proprietary business logic the supplier owns (its secret sauce).

By using confidential containers, the supplier can run its workloads in the customer’s on-prem environment and still protect its business logic from the customer even though the customer has full control of the on-prem environment. And with the use of a confidential container and its use of a CVM allows for further isolation of that workload from the rest of the datacenter infrastructure.

Total tenant isolation / Service provider

From talking to several public organizations, we’ve found that consolidating OpenShift services on a central infrastructure helps solve several problems. This includes reducing the number of administrators and providing dynamic load balancing between the tenants with higher utilization. This does, however, reduce the isolation between the different tenants, which many organizations are not comfortable with. Such organizations require a strict separation between the OpenShift tenants to better prevent data leaks.

With confidential computing we are able to deploy all OpenShift services shared between tenants as confidential objects, helping prevent unauthorized tenants or users from obtaining the confidential data in use. This makes it possible for additional organizations to move to these consolidated services.

Aside from public organizations, specialized service providers (focusing on specific industries such as financial services) are also interested in a similar solution. We have seen use cases where a specialized service provider acts like a public cloud provider for their clients. In order to do this, they need to provide an environment to their customers while maintaining the isolation of each customer and adhering to regulations and laws.

Advanced use cases

We are seeing traction on two additional use cases for CoCo on public cloud and CoCo on-prem: support for confidential GPUs and secure cloud-bursting deployments.

For additional information on those use cases we recommend reading our previous articles on secure cloud bursting and CoCo with confidential GPUs for AI workloads.

CoCo and third party attestation solutions

As we are concentrating on deploying the solution in an on-prem bare metal environment, the importance of remote attestation remains. The confidential compute attestation operator provides the remote attestation functionality. It has built-in support for Intel TDX and AMD SEV-SNP TEEs. Additionally, it can work with external third-party attestation services.

Working with external attestation services

The confidential compute attestation operator can also use external attestation services (AS) supported by Trustee as shown in the following diagram:

As you can see, instead of Trustee providing the attestation service, it’s relying on an external attestation service.

The value that the confidential compute attestation operator brings for such deployments is the abstraction of third-party attestation services from OpenShift confidential containers. The same interfaces are used between the Trustee agent and key broker service (KBS) regardless of the backend attestation service being used.

**Intel Trust Authority for OpenShift attestation **

The Trustee project supports Intel Trust Authority (ITA) providing attestation services for Intel TDX, AMD SEV-SNP and accelerated compute infrastructure such as those provided by NVIDIA. The confidential compute attestation operator now provides support for using ITA in OpenShift clusters:

The following diagram shows how the different components we’ve described come together when using Intel TDX and ITA:

It should be noted that similar to Trustee working with external attestation servers, it can work with external key managers as explained in this article.

Deploying CoCo bare metal

The CoCo bare metal deployment relies on the following parts:

• Assisted Installer for OpenShift—used for deploying OpenShift along with the OpenShift sandboxed containers (OSC) operator which includes OpenShift confidential containers support
• Attestation operator install helper—helper script to install and configure confidential compute attestation operator
• Confidential container install helper—helper script for setting up CoCo on bare metal OpenShift worker nodes using the OSC operator

Assisted installer for installing OSC

Assisted installer for OpenShift is a user-friendly installation solution offered on the Red Hat Hybrid Cloud Console. We leverage it for deploying OSC on bare metal servers.

The following diagram shows how to choose the OpenShift sandboxed containers operator via Assisted installer:

Attestation operator install helper

In the general case, there are a number of deployment considerations that should be addressed when deciding where to deploy the attestation operator:

• How do you bootstrap, verify and trust the TEE in an untrusted environment?
• What are the components of your trusted environment?
• What are the workload (pod) requirements when deployed in the TEE environment?

These questions relate to the topic of Trusted Computing Base (TCB) which includes hardware, firmware and software components for the CoCo solution and should be constructed when deploying the OpenShift CoCo solution. For additional details on this topic, we recommend reading Deployment considerations for Red Hat OpenShift Confidential Containers solution.

For simplicity, when testing and experimenting, we recommend installing this operator on the same bare metal deployment where your OpenShift cluster has been installed.

The install helper script takes care of deploying and configuring the confidential compute attestation operator.

Confidential container install helper

Once your OSC operator and confidential compute attestation operator have been installed, this script will take care everything else you require, including:

• Deploying on Intel TDX machines
• Deploying on AMD SEV-SNP machines
• Etc

Future releases and consolidation on assisted installer

As the CoCo bare metal releases progress, we expect to gradually move all additional steps described here (confidential compute attestation operator and CoCo helper script) into an assisted installer to help simplify deployment.

Current limitations

• Self-signed certificate authority (CA) certificates for the container registry are not supported, so the container registry must use public CA signed certificates. This is important if you are using private registries. Container registries like quay.io, ghcr.io, hub.docker.com, etc. use public CA signed certificates and we recommend using any of these container registries for this release
• There is no support for encrypted container images while signed container images are supported
• Container image double pull—the container image is downloaded and executed inside the confidential VM that executes inside the TEE. Currently, this container image is also downloaded on the worker node

Demo: CoCo bare metal and Red Hat OpenShift AI

Wrap up

In this article we introduced the confidential containers bare metal solution and some of the use cases it addresses, specifically for on-prem environments. We’ve also provided a short overview of how it’s deployed in practice and have shown a video of using the CoCo bare metal for deploying OpenShift AI workloads. In upcoming articles we will provide hands-on instructions for trying out confidential containers on bare metal.