Headline
GHSA-m4hf-6vgr-75r2: K3s apiserver port is vulnerable to unauthenticated remote denial-of-service (DoS) attack via TLS SAN stuffing attack
Impact
An issue was found in K3s where an attacker with network access to K3s servers’ apiserver/supervisor port (TCP 6443) can force the TLS server to add entries to the certificate’s Subject Alternative Name (SAN) list, through a stuffing attack, until the certificate grows so large that it exceeds the maximum size allowed by TLS client implementations. OpenSSL for example will raise an excessive message size
error when this occurs. No authentication is necessary to perform this attack, only the ability to perform a TLS handshake against the apiserver/supervisor port (TCP 6443).
Affected servers will continue to operate, but clients (including both external administrative access with kubectl
and server or agent nodes) will fail to establish new connections, thus leading to a denial of service (DoS) attack.
Remediation
Upgrade to a fixed release:
- v1.28.1+k3s1
- v1.27.5+k3s1
- v1.26.8+k3s1
- v1.25.13+k3s1
- v1.24.17+k3s1
If you are using K3s 1.27 or earlier, you must also add the parameter tls-san-security: true
to the K3s configuration to enable enhanced security for the supervisor’s TLS SAN list. This option defaults to true
starting with K3s 1.28.
Note that this flag changes the behavior of K3s servers. You should ensure that you configure node-external-ip
on servers that will be connected to via an external IP, and add tls-san
entries for any load-balancers or VIP addresses that will be associated with the supervisor port. External IPs and load-balancer/VIP addresses will no longer be added to the supervisor certificate’s SAN list unless explicitly configured.
Mitigation
If you cannot upgrade to a fixed release, the certificate can be “frozen” by running the following command against the cluster:
kubectl annotate secret -n kube-system k3s-serving listener.cattle.io/static=true
⚠️ IMPORTANT CAUTION: Note that this mitigation will prevent the certificate from adding new SAN entries when servers join the cluster, and automatically renewing itself when it is about to expire. If you do this, you should delete the annotation when adding new servers to the cluster, or when the certificate is within 90 days of expiring, so that it can be updated. Once that is done, you can freeze it again.
Affected certificates can be reset by performing the following steps:
- Run
kubectl --server https://localhost:6444 delete secret -n kube-system k3s-serving
- Delete
/var/lib/rancher/k3s/server/tls/dynamic-cert.json
from all servers, and restart thek3s
service.
Background
The K3s apiserver/supervisor listener on port TCP 6443 and uses the rancher/dynamiclistener
library to dynamically generate TLS certificates that contain TLS Subject Alternative Names (SAN) for any host name or IP address requested by a client. This is done to allow servers and external load-balancers to be added to the cluster without the administrator having to explicitly know and configure in advance a fixed list of endpoints that the supervisor may be hosted at.
The library allows the embedding application to configure a callback that is used to filter addresses requested by clients; but this was not previously implemented in K3s.
For more information
If you have any questions or comments about this advisory:
- Reach out to the K3s Security team for security related inquiries.
- Open an issue in the K3s repository.
- Verify with our support matrix and product support lifecycle.
Impact
An issue was found in K3s where an attacker with network access to K3s servers’ apiserver/supervisor port (TCP 6443) can force the TLS server to add entries to the certificate’s Subject Alternative Name (SAN) list, through a stuffing attack, until the certificate grows so large that it exceeds the maximum size allowed by TLS client implementations. OpenSSL for example will raise an excessive message size error when this occurs. No authentication is necessary to perform this attack, only the ability to perform a TLS handshake against the apiserver/supervisor port (TCP 6443).
Affected servers will continue to operate, but clients (including both external administrative access with kubectl and server or agent nodes) will fail to establish new connections, thus leading to a denial of service (DoS) attack.
Remediation
Upgrade to a fixed release:
- v1.28.1+k3s1
- v1.27.5+k3s1
- v1.26.8+k3s1
- v1.25.13+k3s1
- v1.24.17+k3s1
If you are using K3s 1.27 or earlier, you must also add the parameter tls-san-security: true to the K3s configuration to enable enhanced security for the supervisor’s TLS SAN list. This option defaults to true starting with K3s 1.28.
Note that this flag changes the behavior of K3s servers. You should ensure that you configure node-external-ip on servers that will be connected to via an external IP, and add tls-san entries for any load-balancers or VIP addresses that will be associated with the supervisor port. External IPs and load-balancer/VIP addresses will no longer be added to the supervisor certificate’s SAN list unless explicitly configured.
Mitigation
If you cannot upgrade to a fixed release, the certificate can be “frozen” by running the following command against the cluster:
kubectl annotate secret -n kube-system k3s-serving listener.cattle.io/static=true
⚠️ IMPORTANT CAUTION: Note that this mitigation will prevent the certificate from adding new SAN entries when servers join the cluster, and automatically renewing itself when it is about to expire. If you do this, you should delete the annotation when adding new servers to the cluster, or when the certificate is within 90 days of expiring, so that it can be updated. Once that is done, you can freeze it again.
Affected certificates can be reset by performing the following steps:
- Run kubectl --server https://localhost:6444 delete secret -n kube-system k3s-serving
- Delete /var/lib/rancher/k3s/server/tls/dynamic-cert.json from all servers, and restart the k3s service.
Background
The K3s apiserver/supervisor listener on port TCP 6443 and uses the rancher/dynamiclistener library to dynamically generate TLS certificates that contain TLS Subject Alternative Names (SAN) for any host name or IP address requested by a client. This is done to allow servers and external load-balancers to be added to the cluster without the administrator having to explicitly know and configure in advance a fixed list of endpoints that the supervisor may be hosted at.
The library allows the embedding application to configure a callback that is used to filter addresses requested by clients; but this was not previously implemented in K3s.
For more information
If you have any questions or comments about this advisory:
- Reach out to the K3s Security team for security related inquiries.
- Open an issue in the K3s repository.
- Verify with our support matrix and product support lifecycle.
References
- GHSA-m4hf-6vgr-75r2
Related news
An Allocation of Resources Without Limits or Throttling vulnerability in SUSE k3s allows attackers with access to K3s servers' apiserver/supervisor port (TCP 6443) cause denial of service. This issue affects k3s: from v1.24.0 before v1.24.17+k3s1, from v1.25.0 before v1.25.13+k3s1, from v1.26.0 before v1.26.8+k3s1, from sev1.27.0 before v1.27.5+k3s1, from v1.28.0 before v1.28.1+k3s1.