Headline
CVE-2023-30841: ⚠️ Store htpasswd files in Secrets instead of ConfigMaps by lentzi90 · Pull Request #1241 · metal3-io/baremetal-operator
Baremetal Operator (BMO) is a bare metal host provisioning integration for Kubernetes. Prior to version 0.3.0, ironic and ironic-inspector deployed within Baremetal Operator using the included deploy.sh
store their .htpasswd
files as ConfigMaps instead of Secrets. This causes the plain-text username and hashed password to be readable by anyone having a cluster-wide read-access to the management cluster, or access to the management cluster’s Etcd storage. This issue is patched in baremetal-operator PR#1241, and is included in BMO release 0.3.0 onwards. As a workaround, users may modify the kustomizations and redeploy the BMO, or recreate the required ConfigMaps as Secrets per instructions in baremetal-operator PR#1241.
What this PR does / why we need it:
The htpasswd files for Ironic and Inspector contains clear text usernames and hashed passwords so it is better to store them in Secrets.
Depending on how exactly Ironic is deployed this could be a breaking change that requires manual action from the user.
I have tested this with the deploy.sh script and confirmed that it is working. Re-deploying Ironic, with the updated kustomization using the script, automatically creates the new Secrets and configures Ironic and Inspector to use them instead of the ConfigMaps.
Note that the ConfigMaps are not automatically removed. Ideally, the user should remove the ConfigMaps and change the credentials. The ConfigMaps in question are named baremetal-operator-ironic-htpasswd-<random-hash> and baremetal-operator-ironic-inspector-htpasswd-<random-hash> and are located in the baremetal-operator-system Namespace by default.
Note that if the credentials are changed, they must also be updated for BMO. This can be done in the same way by re-deploying using the script.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, …) format, will close the issue(s) when PR gets merged):
Fixes #
Steps for changing ConfigMap to Secret
Before you start, confirm if the environment is affected or not.
This can be done with the following commands:
kubectl -n baremetal-operator-system get deployments.apps baremetal-operator-ironic \ -o jsonpath="{.spec.template.spec.containers[?(@.name == ‘ironic-httpd’)].envFrom}{’\n’}" kubectl -n baremetal-operator-system get deployments.apps baremetal-operator-ironic \ -o jsonpath="{.spec.template.spec.containers[?(@.name == ‘ironic’)].envFrom}{’\n’}" kubectl -n baremetal-operator-system get deployments.apps baremetal-operator-ironic \ -o jsonpath="{.spec.template.spec.containers[?(@.name == ‘ironic-inspector’)].envFrom}{’\n’}"
Affected environments would have configMapRefs for the htpasswd files in the output.
If it instead shows secretRefs for the htpasswd files it is not affected.
Start by listing the current ConfigMaps and Secrets for reference, so it is easy to check which are new and which are old later.
kubectl -n baremetal-operator-system get secrets > old-secrets.txt kubectl -n baremetal-operator-system get configmaps > old-configmaps.txt
Generate new credentials and configuration files, for example like this:
IRONIC_USERNAME="$(tr -dc ‘a-zA-Z0-9’ < /dev/urandom | fold -w 12 | head -n 1)" IRONIC_PASSWORD="$(tr -dc ‘a-zA-Z0-9’ < /dev/urandom | fold -w 12 | head -n 1)" IRONIC_INSPECTOR_USERNAME="$(tr -dc ‘a-zA-Z0-9’ < /dev/urandom | fold -w 12 | head -n 1)" IRONIC_INSPECTOR_PASSWORD="$(tr -dc ‘a-zA-Z0-9’ < /dev/urandom | fold -w 12 | head -n 1)"
echo "IRONIC_HTPASSWD=$(htpasswd -n -b -B “${IRONIC_USERNAME}” “${IRONIC_PASSWORD}”)" > ironic-htpasswd echo "INSPECTOR_HTPASSWD=$(htpasswd -n -b -B “${IRONIC_INSPECTOR_USERNAME}” \ “${IRONIC_INSPECTOR_PASSWORD}”)" > ironic-inspector-htpasswd
cat > ironic-auth-config <<-EOF [ironic] auth_type=http_basic username=${IRONIC_USERNAME} password=${IRONIC_PASSWORD} EOF
cat > ironic-inspector-auth-config <<-EOF [inspector] auth_type=http_basic username=${IRONIC_INSPECTOR_USERNAME} password=${IRONIC_INSPECTOR_PASSWORD} EOF
Create new secrets from htpasswd files, authentication configuration files and credentials for BareMetalOperator.
The exact names of these do not matter and the ones provided here are just examples.
Htpasswd
kubectl -n baremetal-operator-system create secret generic baremetal-operator-ironic-htpasswd \ –from-env-file=ironic-htpasswd kubectl -n baremetal-operator-system create secret generic baremetal-operator-ironic-inspector-htpasswd \ –from-env-file=ironic-inspector-htpasswd
Auth config
kubectl -n baremetal-operator-system create secret generic baremetal-operator-ironic-auth-config \ –from-file=auth-config=ironic-auth-config kubectl -n baremetal-operator-system create secret generic baremetal-operator-ironic-inspector-auth-config \ –from-file=auth-config=ironic-inspector-auth-config
BMO credentials
kubectl -n baremetal-operator-system create secret generic ironic-credentials \ –from-literal=username="${IRONIC_USERNAME}" --from-literal=password="${IRONIC_PASSWORD}" kubectl -n baremetal-operator-system create secret generic ironic-inspector-credentials \ –from-literal=username="${IRONIC_INSPECTOR_USERNAME}" --from-literal=password="${IRONIC_INSPECTOR_PASSWORD}"
Edit the Ironic Deployment to make it use the Secrets instead of the ConfigMaps.
Unfortunately, it is not easy to generate a fool proof patch (for kubectl patch) for this, since the envFrom field is an array.
It is only possible to replace the whole array or do index based replacements, which cannot really be used without knowing the exact details of the specific environment.
For this reason we only show the steps using kubectl edit instead of providing a patch that would likely not work for everyone.
Edit the Deployment with this command: kubectl -n baremetal-operator-system edit deployment baremetal-operator-ironic.
Note that the names of the ConfigMaps may differ depending on version and how it was deployed (e.g. the baremetal-operator- prefix may not be included in v0.1.2).
The order of the containers and other fields can also differ.
The necessary changes are described here:
apiVersion: apps/v1 kind: Deployment metadata: name: baremetal-operator-ironic spec: template: spec: containers: # Only the containers ironic-httpd, ironic and ironic-inspector need changes - name: ironic-httpd envFrom: # Add the secretRefs like this - secretRef: name: baremetal-operator-ironic-htpasswd - secretRef: name: baremetal-operator-ironic-inspector-htpasswd # Remove these old configMapRefs - configMapRef: name: baremetal-operator-ironic-htpasswd-<random-hash> - configMapRef: name: baremetal-operator-ironic-inspector-htpasswd-<random-hash> - name: ironic envFrom: # Add the secretRef like this - secretRef: name: baremetal-operator-ironic-htpasswd # Remove the old configMapRef - configMapRef: name: baremetal-operator-ironic-htpasswd-<random-hash> - name: ironic-inspector envFrom: # Add the secretRef like this - secretRef: name: baremetal-operator-ironic-inspector-htpasswd # Remove the old configMapRef - configMapRef: name: baremetal-operator-ironic-inspector-htpasswd-<random-hash> volumes: # Change the secret name for the ironic-auth-config and ironic-inspector-auth-config # to match what was created earlier. - name: ironic-auth-config secret: defaultMode: 420 secretName: baremetal-operator-ironic-auth-config - name: ironic-inspector-auth-config secret: defaultMode: 420 secretName: baremetal-operator-ironic-inspector-auth-config
Now patch the BareMetalOperator Deployment to make use of the new credentials.
cat > patch.yaml <<-EOF apiVersion: apps/v1 kind: Deployment metadata: name: baremetal-operator-controller-manager spec: template: spec: volumes: - name: ironic-credentials secret: defaultMode: 420 secretName: ironic-credentials - name: ironic-inspector-credentials secret: defaultMode: 420 secretName: ironic-inspector-credentials EOF kubectl -n baremetal-operator-system patch deployment baremetal-operator-controller-manager --patch-file=patch.yaml
Check that the new Pods become Running and then delete the old ConfigMaps and Secrets (optional).
Check that the Pods become Running
kubectl -n baremetal-operator-system get pods
Delete old ConfigMaps (check old-configmaps.txt if you are unsure)
kubectl -n baremetal-operator-system delete configmap baremetal-operator-ironic-htpasswd-<random-hash> kubectl -n baremetal-operator-system delete configmap baremetal-operator-ironic-inspector-htpasswd-<random-hash>
Delete old Secrets (check old-secrets.txt if you are unsure)
kubectl -n baremetal-operator-system delete secret baremetal-operator-ironic-auth-config-<random-hash> kubectl -n baremetal-operator-system delete secret baremetal-operator-ironic-inspector-auth-config-<random-hash> kubectl -n baremetal-operator-system delete secret ironic-credentials-<random-hash> kubectl -n baremetal-operator-system delete secret ironic-inspector-credentials-<random-hash>
Confirm the fix by again checking the output of these commands:
kubectl -n baremetal-operator-system get deployments.apps baremetal-operator-ironic \ -o jsonpath="{.spec.template.spec.containers[?(@.name == ‘ironic-httpd’)].envFrom}{’\n’}" kubectl -n baremetal-operator-system get deployments.apps baremetal-operator-ironic \ -o jsonpath="{.spec.template.spec.containers[?(@.name == ‘ironic’)].envFrom}{’\n’}" kubectl -n baremetal-operator-system get deployments.apps baremetal-operator-ironic \ -o jsonpath="{.spec.template.spec.containers[?(@.name == ‘ironic-inspector’)].envFrom}{’\n’}"
They should now show only secretRefs for htpasswd files.
Clean up temporary files:
rm patch.yaml rm old-secrets.txt rm old-configmaps.txt shred -u ironic-htpasswd shred -u ironic-inspector-htpasswd shred -u ironic-auth-config shred -u ironic-inspector-auth-config
Related news
Red Hat Security Advisory 2023-6143-01 - An update for ztp-site-generate-container, topology-aware-lifecycle-manager and bare-metal-event-relay is now available for Red Hat OpenShift Container Platform 4.14.
Red Hat Security Advisory 2023-1326-01 - Red Hat OpenShift Container Platform is Red Hat's cloud computing Kubernetes application platform solution designed for on-premise or private cloud deployments. This advisory contains the container images for Red Hat OpenShift Container Platform 4.13.0. Issues addressed include bypass, denial of service, information leakage, out of bounds read, and remote SQL injection vulnerabilities.
Red Hat OpenShift Container Platform release 4.13.0 is now available with updates to packages and images that fix several bugs and add enhancements. This release includes a security update for Red Hat OpenShift Container Platform 4.13. Red Hat Product Security has rated this update as having a security impact of Important. A Common Vulnerability Scoring System (CVSS) base score, which gives a detailed severity rating, is available for each vulnerability from the CVE link(s) in the References section.This content is licensed under the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/). If you distribute this content, or a modified version of it, you must provide attribution to Red Hat Inc. and provide a link to the original. Related CVEs: * CVE-2021-4235: A flaw was found in go-yaml. This issue occurs due to unbounded alias chasing, where a maliciously crafted YAML file can cause the system to consume significant system resources. If p...
### Impact Ironic and ironic-inspector deployed within Baremetal Operator using the included `deploy.sh` store their `.htpasswd` files as ConfigMaps instead of Secrets. This causes the plain-text username and hashed password to be readable by anyone having a cluster-wide read-access to the management cluster, or access to the management cluster's Etcd storage. ### Patches This issue is patched in [baremetal-operator PR#1241](https://github.com/metal3-io/baremetal-operator/pull/1241), and is included in BMO release 0.3.0 onwards. ### Workarounds User may modify the kustomizations and redeploy the BMO, or recreate the required ConfigMaps as Secrets per instructions in [baremetal-operator PR#1241](https://github.com/metal3-io/baremetal-operator/pull/1241)