Headline
GHSA-8pgv-569h-w5rw: otelgrpc DoS vulnerability due to unbound cardinality metrics
Summary
The grpc Unary Server Interceptor opentelemetry-go-contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go
// UnaryServerInterceptor returns a grpc.UnaryServerInterceptor suitable
// for use in a grpc.NewServer call.
func UnaryServerInterceptor(opts ...Option) grpc.UnaryServerInterceptor {
out of the box adds labels
net.peer.sock.addrnet.peer.sock.port
that have unbound cardinality. It leads to the server’s potential memory exhaustion when many malicious requests are sent.
Details
An attacker can easily flood the peer address and port for requests.
PoC
Apply the attached patch to the example and run the client multiple times. Observe how each request will create a unique histogram and how the memory consumption increases during it.
Impact
In order to be affected, the program has to configure a metrics pipeline, use UnaryServerInterceptor, and does not filter any client IP address and ports via middleware or proxies, etc.
Others
It is similar to already reported vulnerabilities.
- GHSA-5r5m-65gx-7vrh (open-telemetry/opentelemetry-go-contrib)
- GHSA-cg3q-j54f-5p7p (prometheus/client_golang)
Workaround for affected versions
As a workaround to stop being affected, a view removing the attributes can be used.
The other possibility is to disable grpc metrics instrumentation by passing otelgrpc.WithMeterProvider option with noop.NewMeterProvider.
Solution provided by upgrading
In PR #4322, to be released with v0.46.0, the attributes were removed.
References
Summary
The grpc Unary Server Interceptor opentelemetry-go-contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go
// UnaryServerInterceptor returns a grpc.UnaryServerInterceptor suitable
// for use in a grpc.NewServer call.
func UnaryServerInterceptor(opts ...Option) grpc.UnaryServerInterceptor {
out of the box adds labels
- net.peer.sock.addr
- net.peer.sock.port
that have unbound cardinality. It leads to the server’s potential memory exhaustion when many malicious requests are sent.
Details
An attacker can easily flood the peer address and port for requests.
PoC
Apply the attached patch to the example and run the client multiple times. Observe how each request will create a unique histogram and how the memory consumption increases during it.
Impact
In order to be affected, the program has to configure a metrics pipeline, use UnaryServerInterceptor, and does not filter any client IP address and ports via middleware or proxies, etc.
Others
It is similar to already reported vulnerabilities.
GHSA-rcjv-mgp8-qvmr (open-telemetry/opentelemetry-go-contrib)
GHSA-5r5m-65gx-7vrh (open-telemetry/opentelemetry-go-contrib)
GHSA-cg3q-j54f-5p7p (prometheus/client_golang)
Workaround for affected versions
As a workaround to stop being affected, a view removing the attributes can be used.
The other possibility is to disable grpc metrics instrumentation by passing otelgrpc.WithMeterProvider option with noop.NewMeterProvider.
Solution provided by upgrading
In PR #4322, to be released with v0.46.0, the attributes were removed.
References
- #4322
References
- GHSA-8pgv-569h-w5rw
- https://nvd.nist.gov/vuln/detail/CVE-2023-47108
- open-telemetry/opentelemetry-go-contrib#4322
- open-telemetry/opentelemetry-go-contrib@b44dfc9
- https://github.com/open-telemetry/opentelemetry-go-contrib/blob/9d4eb7e7706038b07d33f83f76afbe13f53d171d/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go#L327
- https://github.com/open-telemetry/opentelemetry-go-contrib/blob/instrumentation/google.golang.org/grpc/otelgrpc/v0.45.0/instrumentation/google.golang.org/grpc/otelgrpc/config.go#L138
- https://pkg.go.dev/go.opentelemetry.io/otel/metric/noop#NewMeterProvider
Related news
Red Hat Security Advisory 2024-6811-03 - Red Hat OpenShift Container Platform release 4.13.51 is now available with updates to packages and images that fix several bugs and add enhancements. Issues addressed include a denial of service vulnerability.
Red Hat Security Advisory 2024-5433-03 - Red Hat OpenShift Container Platform release 4.14.35 is now available with updates to packages and images that fix several bugs and add enhancements. Issues addressed include denial of service and memory exhaustion vulnerabilities.
Red Hat Security Advisory 2024-4591-03 - Updated images that include numerous enhancements, security, and bug fixes are now available for Red Hat OpenShift Data Foundation 4.16.0 on Red Hat Enterprise Linux 9. Issues addressed include denial of service, memory leak, and resource exhaustion vulnerabilities.
Red Hat Security Advisory 2024-0041-03 - Red Hat OpenShift Container Platform release 4.16.0 is now available with updates to packages and images that fix several bugs and add enhancements. Issues addressed include denial of service, memory exhaustion, password leak, and resource exhaustion vulnerabilities.
Red Hat Security Advisory 2024-1891-03 - Red Hat OpenShift Container Platform release 4.14.22 is now available with updates to packages and images that fix several bugs and add enhancements. Issues addressed include cross site scripting, denial of service, and traversal vulnerabilities.
Red Hat Security Advisory 2024-1887-03 - Red Hat OpenShift Container Platform release 4.15.10 is now available with updates to packages and images that fix several bugs and add enhancements. Issues addressed include a denial of service vulnerability.
Red Hat Security Advisory 2024-1812-03 - Custom Metrics Autoscaler Operator for Red Hat OpenShift including security updates. Issues addressed include denial of service and memory leak vulnerabilities.
Red Hat Security Advisory 2024-1458-03 - Red Hat OpenShift Container Platform release 4.14.18 is now available with updates to packages and images that fix several bugs and add enhancements. Issues addressed include a denial of service vulnerability.
Red Hat Security Advisory 2024-1328-03 - Red Hat Advanced Cluster Management for Kubernetes 2.9.3 General Availability release images, which fix bugs and update container images. Issues addressed include denial of service and traversal vulnerabilities.
Red Hat Security Advisory 2024-0766-03 - Red Hat OpenShift Container Platform release 4.15.0 is now available with updates to packages and images that fix several bugs and add enhancements. Issues addressed include a denial of service vulnerability.
Red Hat Security Advisory 2024-0741-03 - Red Hat OpenShift Container Platform release 4.13.33 is now available with updates to packages and images that fix several bugs and add enhancements. Issues addressed include denial of service and traversal vulnerabilities.
Red Hat Security Advisory 2024-0642-03 - An update is now available for Red Hat OpenShift Container Platform 4.14. Issues addressed include denial of service and traversal vulnerabilities.
Red Hat Security Advisory 2024-0489-03 - Red Hat OpenShift Container Platform release 4.12.48 is now available with updates to packages and images that fix several bugs and add enhancements. Issues addressed include a denial of service vulnerability.
Red Hat Security Advisory 2024-0288-03 - Red Hat OpenShift Container Platform release 4.13.30 is now available with updates to packages and images that fix several bugs and add enhancements. Issues addressed include a denial of service vulnerability.
Red Hat Security Advisory 2024-0204-03 - Red Hat OpenShift Container Platform release 4.14.9 is now available with updates to packages and images that fix several bugs and add enhancements. Issues addressed include a denial of service vulnerability.
OpenTelemetry-Go Contrib is a collection of third-party packages for OpenTelemetry-Go. Prior to version 0.46.0, the grpc Unary Server Interceptor out of the box adds labels `net.peer.sock.addr` and `net.peer.sock.port` that have unbound cardinality. It leads to the server's potential memory exhaustion when many malicious requests are sent. An attacker can easily flood the peer address and port for requests. Version 0.46.0 contains a fix for this issue. As a workaround to stop being affected, a view removing the attributes can be used. The other possibility is to disable grpc metrics instrumentation by passing `otelgrpc.WithMeterProvider` option with `noop.NewMeterProvider`.