Confidential containers with AMD SEV

Based on Kata Containers, the Confidential Containers (CoCo) project is a community solution to enable hardware technologies for virtualized memory encryption in container environments through attestation. CoCo SEV enables an encrypted container launch feature by utilizing a remote key broker service to verify the guest measured environment before releasing the image decryption key during orchestration. This blog demonstrates how to prepare an EPYC™ CPU-powered machine for SEV and CoCo, how to install CoCo using a Kubernetes operator, and how to create an encrypted image and start a container pod that uses CoCo SEV.

Requirements

First and foremost, you must have a machine powered by an AMD EPYC™ 7002 Series processor (or later) with Secure Encrypted Virtualization (SEV).

For this blog, I’m using two machines:

• A Dell Inc. PowerEdge R6515 Server running Fedora Linux 35 and a single-node Kubernetes cluster
• A Fedora 36 (x86_64) workstation used to create the encrypted image for the CoCo workload

This separation is not mandatory. You can build encrypted container images on the cluster nodes if you want too.

Server setup

Here are the required packages on the server:

• git
• pip
• mysql-server
• cargo and rustfmt (used to build simple-kbs)

Install the sev-snp-measure tool so you can measure the guest VM:

$ pip3 install sev-snp-measure

Install Kubernetes, and configure it to use the containerd runtime. For this tutorial, I created a single-node Kubernetes v1.24 cluster with the kubeadm tool.

Workstation setup

Creating an encrypted image doesn’t involve specialized encryption hardware, so it can be created on any machine. I used a separate machine for convenience. If you’re using a separate workstation to build your containers, then you need these packages:

• git
• docker-ce
• skopeo
• cargo and rustfmt and openssl-devel (used to build the CoCo Keyprovider)

Create an encrypted container image

CoCo can deal with regular unencrypted container images, but this blog focuses on a confidential pod with an encrypted image. This section describes how to create a custom nginx image, encrypt it with Skopeo, and finally push it to a Docker Hub registry for later use as a pod image.

I’m using Docker Hub because it supports storing encrypted images. If you use a different registry, make sure it supports encrypted images.

1. Create an image

First, create a Dockerfile to define a custom NGINX image with a special HTML page:

FROM nginx:latest

RUN echo "Welcome to Confidential Containers World!" >> /usr/share/nginx/html/coco.html

Build it:

$ docker build -t coco-custom-nginx .

You’ve now created an unencrypted image called coco-custom-nginx in your local Docker storage.

2. Start the CoCo keyprovider

Skopeo uses the ocicrypt library to encrypt a container image. An encryption key is provided to Skopeo through the CoCo Keyprovider. This is a key provider implementation in the attestation-agent project. Build and start the key provider on localhost, port 50001:

$ git clone https://github.com/containers/attestation-agent
$ cd attestation-agent
$ git checkout v0.5.0 -b v0.5.0
$ cd coco_keyprovider
$ cargo build --release
$ cp target/release/coco_keyprovider .
$ RUST_LOG=coco_keyprovider \
./coco_keyprovider --socket 127.0.0.1:50001 &

The ocicrypt.conf file contains connection information for the key providers. Create the configuration file and export the OCICRYPT_KEYPROVIDER_CONFIG variable:

$ cat <<EOF > ocicrypt.conf
{
"key-providers": {
  "attestation-agent": {
  "grpc": "127.0.0.1:50001"
  }}}
EOF
$ export OCICRYPT_KEYPROVIDER_CONFIG="$(pwd)/ocicrypt.conf"

3. Generate an encryption key

The encryption key must have a length of 32 bytes. Generate a random base64 encoded key of the appropriate length:

$ export ENC_KEY_BASE64=$(head -c32 < /dev/random | base64)
$ echo $ENC_KEY_BASE64
hRF3FrDC1LBpWtgVpXL3GyIgxNXGG1f5q4HdeR5kDZU=

Copy the encryption key to a file and export the key file path to the ENC_KEY_FILE variable:

$ echo $ENC_KEY_BASE64 | base64 -d > key1
$ export ENC_KEY_FILE="$PWD/key1"

4. Encrypt the image

Encrypt the coco-custom-nginx image from your local Docker storage, and then push it to the registry (in this example, wainersm/coco-custom-nginx:encrypted). The new registry must be created in Docker Hub with your user account.

$ docker login -u <your-username>
$ skopeo copy --insecure-policy --encryption-key \
provider:attestation-agent:keypath=${ENC_KEY_FILE}::keyid=kbs:///default/key/key_id1
docker-daemon:coco-custom-nginx:latest \
docker://<your-username>/coco-custom-nginx:encrypted

Getting image source signatures
...
Copying config 9ef34dd43c done
Writing manifest to image destination
Storing signatures

The skopeo copy command can use several different transport protocols to copy from origin to the destination image. In this example, I’ve copied the source image from my local Docker storage using the docker-daemon protocol.

Currently, Skopeo is not able to encrypt an image to local Docker storage. It silently fails, so copy the image directly to the registry, and then verify that it was actually encrypted:

$ skopeo inspect \
docker://<your-username>/coco-custom-nginx:encrypted | \
jq '.LayersData\[\].MIMEType'

"application/vnd.oci.image.layer.v1.tar+gzip+encrypted"
"application/vnd.oci.image.layer.v1.tar+gzip+encrypted"
...
"application/vnd.oci.image.layer.v1.tar+gzip+encrypted"
"application/vnd.oci.image.layer.v1.tar+gzip+encrypted"

From the output, you can see that the mimetypes indicate the layers are indeed encrypted.

Prepare the SEV host system

Secure Memory Encryption (SME) needs to be enabled in the system BIOS settings. Using a PowerEdge R6515 server as an example, the Dell Remote Access Controller (DRAC) can be configured to enable this setting in Configuration > BIOS Settings > Processor Settings > Secure Memory Encryption - Enabled

For SEV support, your host system must have a Linux kernel version 5.11 or later. Ensure the running Kernel Virtual Machine (KVM) module has SEV enabled by adding the kvm_amd.sev=1 parameter to the system kernel command line in the GRUB configuration:

$ sudo grubby --default-kernel \
/boot/vmlinuz-5.16.11-200.fc35.x86_64
$ sudo grubby --args="kvm_amd.sev=1" --update-kernel \
/boot/vmlinuz-5.16.11-200.fc35.x86_64
$ sudo grubby --info /boot/vmlinuz-5.16.11-200.fc35.x86_64
index=4
kernel="/boot/vmlinuz-5.16.11-200.fc35.x86_64"
args="ro rootflags=subvol=root ro rootflags=subvol=root
console=ttyS1,115200 kvm_amd.sev=1"
root="UUID=5304c86d-73a8-44b8-9323-fd4a7e52f75b"
initrd="/boot/initramfs-5.16.11-200.fc35.x86_64.img"
title="Linux 5.16.11-200.fc35.x86_64"
id="a6fa0c7913f646b2a3b842ccfe8e8298-5.16.11-200.fc35.x86_64"

Reboot the machine and verify that the messages in the Kernel ring buffer mention SEV:

$ sudo dmesg | grep -i sev
[ 0.000000] Command line:
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.16.11-200.fc35.x86_64
root=UUID=5304c86d-73a8-44b8-9323-fd4a7e52f75b ro rootflags=subvol=root
ro rootflags=subvol=root console=ttyS1,115200 kvm_amd.sev=1
[ 0.069214] Kernel command line:
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.16.11-200.fc35.x86_64
root=UUID=5304c86d-73a8-44b8-9323-fd4a7e52f75b ro rootflags=subvol=root
ro rootflags=subvol=root console=ttyS1,115200 kvm_amd.sev=1
[ 5.754753] ccp 0000:42:00.1: sev enabled
[ 5.811254] ccp 0000:42:00.1: SEV firmware update successful
[ 6.142119] ccp 0000:42:00.1: SEV API:1.42 build:42
[ 19.973676] SEV supported: 509 ASIDs

1. Generate the certificate chain

The SEV certificate chain file contains the AMD Root Key (ARK) and the AMD SEV Signing Key (ASK) certificates. This certificate chain must be downloaded and saved onto the host by an appropriate authority so that the measurement can be verified during launch attestation.

Use the sevctl tool to download the certificate chain file.

At the time of writing, the Fedora 35 package of sevctl had a bug that caused the certificate chain file creation to fail. Build the tool from source:

$ git clone https://github.com/virtee/sevctl
$ cd sevctl
$ cargo build --release
$ cp target/release/sevctl .
$ sudo mkdir -p /opt/sev/
$ sudo ./sevctl export --full /opt/sev/cert_chain.cert

2. Install CoCo

The CoCo runtime is bundled in a Kubernetes operator you can deploy on your cluster. You must install the CoCo operator version 0.5.0 or higher.

The cluster admin uses labels to instruct the operator controller about which nodes, in a multi-node cluster, need the runtime. You must have the node-role.kubernetes.io/worker= label on all cluster nodes that you want the runtime installed on:

# kubectl label node "$(hostname)" "node-role.kubernetes.io/worker="
node/virtlab1012 labeled
# kubectl get nodes
NAME        STATUS ROLES                AGE VERSION
virtlab1012 Ready  control-plane,worker 24m v1.24.0

Once the target worker nodes are properly labeled, the next step is to install the operator controller. For this step, ensure that SELinux is in permissive mode. The operator controller attempts to restart services on your system that SELinux may deny. Use the following sequence of commands to install the operator controller:

# kubectl apply -k \
github.com/confidential-containers/operator/config/release?ref=v0.5.0

This creates a series of resources in the confidential-containers-system namespace. In particular, it creates a deployment with pods, all of which need to be running before you continue the installation.

# kubectl get pods -n confidential-containers-system
NAME                                READY STATUS...
cc-operator-controller-manager...5jtsk 2/2   Running...

The operator controller is capable of managing the installation of different CoCo runtimes through Kubernetes custom resources. You must install the runtime that provides the support for AMD SEV:

# kubectl apply -k \
github.com/confidential-containers/operator/config/samples/ccruntime/default?ref=v0.5.0

Next, ensure that all the pods in the confidential-containers-system namespace are running and the system has the kata-qemu-sev runtime class configured in the cluster:

# kubectl get pods -n confidential-containers-system

NAME                                READY STATUS...
cc-operator-controller-manager...5jtsk 2/2   Running...
cc-operator-daemon-install-hvfs7    1/1   Running...
cc-operator-pre-install-daemon-4jhkn    1/1  Running...

# kubectl get runtimeclass
NAME        HANDLER     AGE
kata        kata        4m21s
kata-clh    kata-clh    4m21s
kata-clh-tdx  kata-clh-tdx  4m21s
kata-qemu   kata-qemu   4m21s
kata-qemu-sev kata-qemu-sev 4m21s
kata-qemu-tdx kata-qemu-tdx 4m21s

It’s a good idea to create a simple Kata Containers pod to ensure that the installation is working as expected before moving on to the next section. Follow the instructions in the project quickstart.

Configure the Key Broker Server (KBS)

The Confidential Containers simple-kbs is a basic prototype key broker that can validate a guest measurement according to a specified policy. By evaluating these policies, a secret or key can be conditionally released. In this case, the image decryption key is released based on a policy that defines an expected SEV launch measurement. If the measured value during launch matches this expected value in the simple-kbs, the image decryption key is released, the image is decrypted and the launch of the pod proceeds.

Clone and build the simple-kbs:

$ git clone https://github.com/confidential-containers/simple-kbs.git
$ cd simple-kbs
$ cargo build --release
$ cp target/release/simple-kbs .

1. Initialize the simple-kbs database

The simple-kbs relies on a SQL database to store its secrets and policies. It currently supports SQLite, Postgres, and MySQL. For this tutorial, I use a MySQL database.

To set up and install the simple-kbs, environment variables must be specified detailing the database connection information and authentication details:

• KBS_DB_TYPE: Database type
• KBS_DB_HOST: Database address (host:port)
• KBS_DB_USER: Database username
• KBS_DB_PW: Database password
• KBS_DB: Database name

For example:

$ export KBS_DB_TYPE=mysql
$ export KBS_DB_HOST=localhost
$ export KBS_DB_USER=root
$ export KBS_DB_PW=My_Strong_Passphrase
$ export KBS_DB=sev_attest

Export the encryption key (encoded in base64) used to encrypt the image in the previous steps:

$ export ENC_KEY="hRF3FrDC1LBpWtgVpXL3GyIgxNXGG1f5q4HdeR5kDZU="

The database can be initialized as follows:

$ mysql -u ${KBS_DB_USER} -p${KBS_DB_PW} -e "CREATE DATABASE ${KBS_DB};"
$ mysql -u ${KBS_DB_USER} -p${KBS_DB_PW} -D ${KBS_DB} < db/db-mysql.sql
$ mysql -u ${KBS_DB_USER} -p${KBS_DB_PW} -D ${KBS_DB} -e "INSERT INTO secrets VALUES (10, 'default/key/key_id1', '${ENC_KEY}', NULL);"

The encryption key (${ENC_KEY}) used to create the docker.io/<your-username>/coco-custom-nginx:encrypted image is stored in the secrets table.

Verify the secret table:

$ mysql -u ${KBS_DB_USER} -p${KBS_DB_PW} -D ${KBS_DB} -e "SELECT* FROM secrets;"
mysql: [Warning] Using a password on the command-line interface can be insecure.

+----+---------------------+------------------+-------+
| id | secret_id           | secret           | polid |
+----+---------------------+------------------+-------+
| 10 | default/key/key_id1 | hRF3Fr...R5kDZU= | NULL  |
+----+---------------------+------------------+-------+

Now the database is configured with the minimal policy for simple-kbs to serve the encryption key.

2. Start the simple-kbs

Start simple-kbs on localhost at port 44444:

$ ./simple-kbs --grpc_sock=0.0.0.0:44444 &

SEV runtime configuration

When you deployed the CoCo operator, several Kubernetes Runtime Classes and their Kata configuration files were installed:

$ ls /opt/confidential-containers/share/defaults/kata-containers
configuration-clh-tdx.toml configuration-qemu-se.toml
configuration-qemu-tdx.toml configuration.toml
configuration-clh.toml configuration-qemu-sev.toml
configuration-qemu.toml
configuration-dragonball.toml configuration-qemu-snp.toml
configuration-remote.toml

The kata-qemu-sev runtime class must be specified to launch the pod with the encrypted image with SEV launch attestation, and its configuration file is configuration-qemu-sev.toml This contains properties that Kata Containers use to instantiate and manage the pod. For example, if the confidential_guest property is set, then the SEV feature for VM memory encryption is active.

The sev_cert_chain property sets the location of the SEV certificate chain that was created in a previous section of this blog.

Run your first confidential pod

For ease of use and demonstration, I’m configuring the simple-kbs with a permissive policy to release the encryption key and allow a container pod to be launched without attestation. To configure the full secure policy, skip to the next section. This section describes how to create a confidential pod without attestation.

Here’s my example pod YAML (coco-sev-nginx.yaml). Replace <your-username> and <server-IP>:

$ cat coco-sev-nginx.yaml
apiVersion: v1
kind: Pod
metadata:
  name: coco-custom-nginx
  annotations:
    io.katacontainers.config.pre_attestation.enabled: "true"
    io.katacontainers.config.pre_attestation.uri: "<server-IP>:44444"
    io.katacontainers.config.sev.policy: "3"
spec:
  containers:
  - name: nginx
    image: <your-username>/coco-custom-nginx:encrypted
    ports:
    - containerPort: 80
  dnsPolicy: ClusterFirst
  runtimeClassName: kata-qemu-sev

Apply the configuration:

$ kubectl apply -f coco-sev-nginx.yaml
pod/coco-custom-nginx created

The runtime class must be specified as kata-qemu-sev. The encrypted image URL is set to <your-username>/coco-custom-nginx:encrypted. Several kubernetes annotations are set to provide the required configuration:

• io.katacontainers.config.pre_attestation.enabled enables SEV launch attestation
• io.katacontainers.config.pre_attestation.uri is the address of the simple-kbs (the system’s IP, in my example)
• io.katacontainers.config.sev.policy is the hardware policy (3 for SEV and 7 for SEV-ES)

Start the pod:

$ kubectl apply -f coco-sev-nginx.yaml

After a little time, verify that the pod is running:

$ kubectl get pods
NAME            READY STATUS  RESTARTS AGE
coco-custom-nginx 1/1   Running 0       3m59s

You’ve just created your first confidential pod using AMD SEV!

Use the guest measurement and verification policy

In the previous section, the minimal simple-kbs policy allowed for release of the encryption key without attestation. This was done as a quick proof-of-concept, and is inherently insecure.

To correctly benefit from the workload encryption key, you must create a policy requiring the guest launch measurement to be verified before release.

When the pod launch is scheduled, the Kata shim layer retrieves the SEV launch measurement from the AMD Platform Security Processor (PSP) and passes it to simple-kbs to be verified against your policy. Data sent to simple-kbs is secured by cryptographic keys and signatures from known, trusted authorities.

The sev-snp-measure tool can be used to recreate the launch measurement. This measurement includes digest values of the guest firmware, kernel, initrd, and kernel command-line parameters.

Set the paths to the necessary files:

$ PATH=$PATH:/opt/confidential-containers/bin/
$ KATA_SEV_CONFIG="/opt/confidential-containers/share/defaults/kata-containers/configuration-qemu-sev.toml"
$ KERNEL_PATH=$(kata-runtime -config "$KATA_SEV_CONFIG" kata-env --json | jq -r '.Kernel.Path')
$ INITRD_PATH=$(kata-runtime -config "$KATA_SEV_CONFIG" kata-env --json | jq -r '.Initrd.Path')
$ FIRMWARE_PATH=$(grep "^firmware = " $KATA_SEV_CONFIG | sed -e 's/.*"\\.*\\".*/\1/')

The kernel command-line parameters are a bit tricky to obtain because Kata combines the values from the configuration file (configuration-qemu-sev.toml) with some additional values generated at runtime. One way to obtain it is to grep for the -append parameter passed to QEMU from the confidential pod youcreated in the previous section:

$ ps aux | grep /opt/confidential-containers/bin/qemu-system-x86_64
root 2656095 99.8 6.7 2565268 21342 ? Sl 14:09 94:39
/opt/confidential-containers/bin/qemu-system-x86_64 -name
...
-kernel
/opt/confidential-containers/share/kata-containers/vmlinuz-5.19.2-102cc-sev
-initrd
/opt/confidential-containers/share/kata-containers/kata-ubuntu-20.04-sev.initrd
-append tsc=reliable no_timer_check rcupdate.rcu_expedited=1
i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp
reboot=k cryptomgr.notests net.ifnames=0 pci=lastbus=0 console=hvc0
console=hvc1 quiet panic=1 nr_cpus=1 selinux=0
agent.aa_kbc_params=online_sev_kbc::10.8.0.194:44444
scsi_mod.scan=none agent.config_file=/etc/agent-config.toml
agent.enable_signature_verification=false
-pidfile
/run/vc/vm/81a554805f4d82ba05133edbba2be5099237a30527334cc33bde8d38c35359e1/pid
-smp 1,cores=1,threads=1,sockets=1,maxcpus=1

The significant portion of that output is the section after -append but before the next option (-pidfile in this example). Export that into an environment variable:

$ export KERNEL_APPEND="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k cryptomgr.notests net.ifnames=0 pci=lastbus=0 console=hvc0 console=hvc1 quiet panic=1 nr_cpus=1 selinux=0 agent.aa_kbc_params=online_sev_kbc::10.8.0.194:44444 scsi_mod.scan=none agent.config_file=/etc/agent-config.toml agent.enable_signature_verification=false"

Use sev-snp-measure to create the guest launch measurement. Then create a launch measurement verification policy by inserting the measurement into the simple-kbs’s database:

$ MEASUREMENT=$(sev-snp-measure --mode=sev --output-format=base64 \
--ovmf="$FIRMWARE_PATH" --kernel="$KERNEL_PATH" \
--initrd="$INITRD_PATH" --append="${KERNEL_APPEND}")
$ echo $MEASUREMENT
tc9SbQPHZYklGwBgI0a/xh5avxeOakZ/AODXnDiWBpQ=
$ mysql -u ${KBS_DB_USER} -p${KBS_DB_PW} -D ${KBS_DB} \
-e "INSERT INTO policy VALUES (10, '\[\\${MEASUREMENT}\\\]', '\[\]', 0, 0, '\[\]', now(), NULL, 1)"
$ mysql -u ${KBS_DB_USER} -p${KBS_DB_PW} -D ${KBS_DB} \
-e "UPDATE secrets SET polid = '10' WHERE secret_id = 'default/key/key_id1';"

Delete and restart the pod to apply the guest measurement policy. To get more detailed information on how simple-kbs responds to the attestation-agent requests, restart the simple-kbs with debugging messages enabled (RUST_LOG=simple_kbs):

$ kubectl delete pod/coco-custom-nginx
pod "coco-custom-nginx" deleted
$ cd simple-kbs
$ RUST_LOG=simple_kbs ./simple-kbs --grpc_sock=0.0.0.0:44444 &
$ cd ..
$ kubectl apply -f coco-sev-nginx.yaml
pod/coco-custom-nginx created
$ kubectl get pods
NAME            READY STATUS  RESTARTS AGE
coco-custom-nginx 1/1   Running 0       5m16s

You can see in simple-kbs logs that the measured digest (tc9SbQPHZYklGwBgI0a/xh5avxeOakZ/AODXnDiWBpQ=) corresponds to expected value from the database:

[2023-05-04T13:11:15Z INFO simple_kbs::grpc] Policy validated
succesfully. Connection: Connection { policy: 3, fw_api_major: 1,
fw_api_minor: 42, fw_build_id: 42, launch_description: "shim
launch", fw_digest: "tc9SbQPHZYklGwBgI0a/xh5avxeOakZ/AODXnDiWBpQ=" }

Encrypted containers

In this article, I’ve demonstrated how to create Confidential Containers using EPYC™ hardware with AMD SEV. Enabling and using SEV defends the memory space utilized by the workload through hardware encryption. In addition to this defense, the launch measurement provided by SEV allows for a policy-based key release to the orchestrator scheduler for decrypting a container image. These methods showcase how the industry is moving towards cryptographic solutions to protect cloud data and infrastructure.

For more details on the CoCo project, read the What is the Confidential Containers project. To understand the topic of measurements, read Understanding the Confidential Containers Attestation Flow.

I would like to acknowledge and thank Beraldo Leal for reviewing this article, and for going through the steps to install and configure Confidential Containers on his AMD SEV machine.