Headline
GHSA-fcm2-6c3h-pg6j: Node DOS by way of memory exhaustion through ExecSync request in CRI-O
Description
An ExecSync request runs a command in a container and returns the output to the Kubelet. It is used for readiness and liveness probes within a pod. The way CRI-O runs ExecSync commands is through conmon. CRI-O asks conmon to start the process, and conmon writes the output to disk. CRI-O then reads the output and returns it to the Kubelet.
If the output of the command is large enough, it is possible to exhaust the memory (or disk usage) of the node. The following deployment is an example yaml file that will output around 8GB of ‘A’ characters, which would be written to disk by conmon and read by CRI-O.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment100
spec:
selector:
matchLabels:
app: nginx
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "seq 1 50000000`; do echo -n 'aaaaaaaaaaaaaaaa'; done"]
Impact
It is possible for the node to be exhausted of memory or disk space, depending on the node the command is being run on. What is further problematic is that the memory and disk usage aren’t attributed to the container, as this file and its processing are implementation details of CRI-O. The consequence of the exhaustion is that other services on the node, e.g. other containers, will be unable to allocate memory and thus causing a denial of service.
Patches
This vulnerability will be fixed in 1.24.1, 1.23.3, 1.22.5, v1.21.8, v1.20.8, v1.19.7
Workarounds
At the time of writing, no workaround exists other than ensuring only trusted images are used.
References
https://github.com/containerd/containerd/security/advisories/GHSA-5ffw-gxpp-mxpf
For more information
If you have any questions or comments about this advisory:
- Open an issue in the CRI-O repo
- To make a report, email your vulnerability to the private [email protected] list with the security details and the details expected for all CRI-O bug reports.
Credits
Disclosed by Ada Logics in a security audit sponsored by CNCF and facilitated by OSTIF.
Description
An ExecSync request runs a command in a container and returns the output to the Kubelet. It is used for readiness and liveness probes within a pod. The way CRI-O runs ExecSync commands is through conmon. CRI-O asks conmon to start the process, and conmon writes the output to disk. CRI-O then reads the output and returns it to the Kubelet.
If the output of the command is large enough, it is possible to exhaust the memory (or disk usage) of the node. The following deployment is an example yaml file that will output around 8GB of ‘A’ characters, which would be written to disk by conmon and read by CRI-O.
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment100 spec: selector: matchLabels: app: nginx replicas: 2 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 lifecycle: postStart: exec: command: ["/bin/sh", "-c", “seq 1 50000000`; do echo -n 'aaaaaaaaaaaaaaaa’; done”]
Impact
It is possible for the node to be exhausted of memory or disk space, depending on the node the command is being run on. What is further problematic is that the memory and disk usage aren’t attributed to the container, as this file and its processing are implementation details of CRI-O. The consequence of the exhaustion is that other services on the node, e.g. other containers, will be unable to allocate memory and thus causing a denial of service.
Patches
This vulnerability will be fixed in 1.24.1, 1.23.3, 1.22.5, v1.21.8, v1.20.8, v1.19.7
Workarounds
At the time of writing, no workaround exists other than ensuring only trusted images are used.
References
GHSA-5ffw-gxpp-mxpf
For more information
If you have any questions or comments about this advisory:
- Open an issue in the CRI-O repo
- To make a report, email your vulnerability to the private
[email protected] list
with the security details and the details expected for all CRI-O bug
reports.
Credits
Disclosed by Ada Logics in a security audit sponsored by CNCF and facilitated by OSTIF.
References
- GHSA-fcm2-6c3h-pg6j
Related news
Red Hat Security Advisory 2022-7457-01 - The container-tools module contains tools for working with containers, notably podman, buildah, skopeo, and runc. Issues addressed include information leakage and memory exhaustion vulnerabilities.
An update for the container-tools:4.0 module is now available for Red Hat Enterprise Linux 8. Red Hat Product Security has rated this update as having a security impact of Moderate. A Common Vulnerability Scoring System (CVSS) base score, which gives a detailed severity rating, is available for each vulnerability from the CVE link(s) in the References section.This content is licensed under the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/). If you distribute this content, or a modified version of it, you must provide attribution to Red Hat Inc. and provide a link to the original. Related CVEs: * CVE-2022-1708: cri-o: memory exhaustion on the node when access to the kube api * CVE-2022-27191: golang: crash in a golang.org/x/crypto/ssh server * CVE-2022-29162: runc: incorrect handling of inheritable capabilities
Red Hat Security Advisory 2022-5392-01 - Red Hat Advanced Cluster Management for Kubernetes 2.3.11 images Red Hat Advanced Cluster Management for Kubernetes provides the capabilities to address common challenges that administrators and site reliability engineers face as they work across a range of public and private cloud environments. Clusters and applications are all visible and managed from a single console—with security policy built in. This advisory contains the container images for Red Hat Advanced Cluster Management for Kubernetes, which resolve security issues and fix several bugs. Issues addressed include a traversal vulnerability.
Red Hat Advanced Cluster Management for Kubernetes 2.3.11 general availability release images, which provide security updates and bug fixes. Red Hat Product Security has rated this update as having a security impact of Moderate. A Common Vulnerability Scoring System (CVSS) base score, which gives a detailed severity rating, is available for each vulnerability from the CVE links in the References section.This content is licensed under the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/). If you distribute this content, or a modified version of it, you must provide attribution to Red Hat Inc. and provide a link to the original. Related CVEs: * CVE-2022-0235: node-fetch: exposure of sensitive information to an unauthorized actor * CVE-2022-0536: follow-redirects: Exposure of Sensitive Information via Authorization Header leak * CVE-2022-21803: nconf: Prototype pollution in memory store * CVE-2022-23806: golang: crypto/elliptic IsOnCurv...
Red Hat OpenShift Container Platform release 3.11.715 is now available with updates to packages and images that fix several bugs and add enhancements. This release includes a security update for Red Hat OpenShift Container Platform 3.11. Red Hat Product Security has rated this update as having a security impact of Moderate. A Common Vulnerability Scoring System (CVSS) base score, which gives a detailed severity rating, is available for each vulnerability from the CVE link(s) in the References section.This content is licensed under the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/). If you distribute this content, or a modified version of it, you must provide attribution to Red Hat Inc. and provide a link to the original. Related CVEs: * CVE-2022-1708: cri-o: memory exhaustion on the node when access to the kube api
Red Hat Security Advisory 2022-4947-01 - Red Hat OpenShift Container Platform is Red Hat's cloud computing Kubernetes application platform solution designed for on-premise or private cloud deployments. This advisory contains the RPM packages for Red Hat OpenShift Container Platform 4.6.59. Issues addressed include cross site scripting and memory exhaustion vulnerabilities.
Red Hat Security Advisory 2022-4951-01 - Red Hat OpenShift Container Platform is Red Hat's cloud computing Kubernetes application platform solution designed for on-premise or private cloud deployments. This advisory contains the RPM packages for Red Hat OpenShift Container Platform 4.8.43. Issues addressed include a memory exhaustion vulnerability.
Red Hat Security Advisory 2022-4965-01 - Red Hat OpenShift Container Platform is Red Hat's cloud computing Kubernetes application platform solution designed for on-premise or private cloud deployments. This advisory contains the RPM packages for Red Hat OpenShift Container Platform 4.7.53. There are no images for this advisory. Issues addressed include a memory exhaustion vulnerability.
Red Hat Security Advisory 2022-4943-01 - Red Hat OpenShift Container Platform is Red Hat's cloud computing Kubernetes application platform solution designed for on-premise or private cloud deployments. This advisory contains the RPM packages for Red Hat OpenShift Container Platform 4.10.18. Issues addressed include a memory exhaustion vulnerability.
Red Hat Security Advisory 2022-4972-01 - Red Hat OpenShift Container Platform is Red Hat's cloud computing Kubernetes application platform solution designed for on-premise or private cloud deployments. This advisory contains the RPM packages for Red Hat OpenShift Container Platform 4.9.38. Issues addressed include a memory exhaustion vulnerability.
Red Hat OpenShift Container Platform release 4.6.59 is now available with updates to packages and images that fix several bugs and add enhancements. This release includes a security update for Red Hat OpenShift Container Platform 4.6. Red Hat Product Security has rated this update as having a security impact of Important. A Common Vulnerability Scoring System (CVSS) base score, which gives a detailed severity rating, is available for each vulnerability from the CVE link(s) in the References section.This content is licensed under the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/). If you distribute this content, or a modified version of it, you must provide attribution to Red Hat Inc. and provide a link to the original. Related CVEs: * CVE-2022-1708: cri-o: memory exhaustion on the node when access to the kube api * CVE-2022-29036: credentials: Stored XSS vulnerabilities in jenkins plugin * CVE-2022-29046: subversion: Stored XSS vu...
Red Hat OpenShift Container Platform release 4.8.43 is now available with updates to packages and images that fix several bugs and add enhancements. This release includes a security update for Red Hat OpenShift Container Platform 4.8. Red Hat Product Security has rated this update as having a security impact of Moderate. A Common Vulnerability Scoring System (CVSS) base score, which gives a detailed severity rating, is available for each vulnerability from the CVE link(s) in the References section.This content is licensed under the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/). If you distribute this content, or a modified version of it, you must provide attribution to Red Hat Inc. and provide a link to the original. Related CVEs: * CVE-2022-1708: cri-o: memory exhaustion on the node when access to the kube api
Red Hat OpenShift Container Platform release 4.7.53 is now available with updates to packages and images that fix several bugs and add enhancements. This release includes a security update for Red Hat OpenShift Container Platform 4.7. Red Hat Product Security has rated this update as having a security impact of Moderate. A Common Vulnerability Scoring System (CVSS) base score, which gives a detailed severity rating, is available for each vulnerability from the CVE link(s) in the References section.This content is licensed under the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/). If you distribute this content, or a modified version of it, you must provide attribution to Red Hat Inc. and provide a link to the original. Related CVEs: * CVE-2022-1708: cri-o: memory exhaustion on the node when access to the kube api
Red Hat OpenShift Container Platform release 4.9.38 is now available with updates to packages and images that fix several bugs and add enhancements. This release includes a security update for Red Hat OpenShift Container Platform 4.9. Red Hat Product Security has rated this update as having a security impact of Moderate. A Common Vulnerability Scoring System (CVSS) base score, which gives a detailed severity rating, is available for each vulnerability from the CVE link(s) in the References section.This content is licensed under the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/). If you distribute this content, or a modified version of it, you must provide attribution to Red Hat Inc. and provide a link to the original. Related CVEs: * CVE-2022-1708: cri-o: memory exhaustion on the node when access to the kube api
Red Hat OpenShift Container Platform release 4.10.18 is now available with updates to packages and images that fix several bugs and add enhancements. This release includes a security update for Red Hat OpenShift Container Platform 4.10. Red Hat Product Security has rated this update as having a security impact of Moderate. A Common Vulnerability Scoring System (CVSS) base score, which gives a detailed severity rating, is available for each vulnerability from the CVE link(s) in the References section.This content is licensed under the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/). If you distribute this content, or a modified version of it, you must provide attribution to Red Hat Inc. and provide a link to the original. Related CVEs: * CVE-2022-1708: cri-o: memory exhaustion on the node when access to the kube api
A vulnerability was found in CRI-O that causes memory or disk space exhaustion on the node for anyone with access to the Kube API. The ExecSync request runs commands in a container and logs the output of the command. This output is then read by CRI-O after command execution, and it is read in a manner where the entire file corresponding to the output of the command is read in. Thus, if the output of the command is large it is possible to exhaust the memory or the disk space of the node when CRI-O reads the output of the command. The highest threat from this vulnerability is system availability.