Headline
CVE-2023-27488: gRPC client produces invalid protobuf when an HTTP header with non-UTF8 value is received.
Envoy is an open source edge and service proxy designed for cloud-native applications. Prior to versions 1.26.0, 1.25.3, 1.24.4, 1.23.6, and 1.22.9, escalation of privileges is possible when failure_mode_allow: true is configured for ext_authz filter. For affected components that are used for logging and/or visibility, requests may not be logged by the receiving service. When Envoy was configured to use ext_authz, ext_proc, tap, ratelimit filters, and grpc access log service and an http header with non-UTF-8 data was received, Envoy would generate an invalid protobuf message and send it to the configured service. The receiving service would typically generate an error when decoding the protobuf message. For ext_authz that was configured with failure_mode_allow: true, the request would have been allowed in this case. For the other services, this could have resulted in other unforeseen errors such as a lack of visibility into requests. As of versions 1.26.0, 1.25.3, 1.24.4, 1.23.6, and 1.22.9, Envoy by default sanitizes the values sent in gRPC service calls to be valid UTF-8, replacing data that is not valid UTF-8 with a ! character. This behavioral change can be temporarily reverted by setting runtime guard envoy.reloadable_features.service_sanitize_non_utf8_strings to false. As a workaround, one may set failure_mode_allow: false for ext_authz.
Impact
Escalation of Privileges when failure_mode_allow: true is configured for ext_authz filter. For affected components that are used for logging and/or visibility, requests may not be logged by the receiving service.
Affected components
ext_authz, ext_proc, tap, ratelimit filters and gRPC access log service.
Attack vector/s
The attacker can use this vulnerability to bypass auth checks when ext_authz is used.
Description
When Envoy was configured to use ext_authz, ext_proc, tap, ratelimit filters, and grpc access log service and an http header with non-UTF-8 data was received, Envoy would generate an invalid protobuf message and send it to the configured service. The receiving service would typically generate an error when decoding the protobuf message.
For ext_authz that was configured with failure_mode_allow: true, the request would have been allowed in this case.
For the other services, this could have resulted in other unforseen errors such as a lack of visibility into requests (eg request not logged).
Envoy will now by default sanitize the values sent in gRPC service calls to be valid UTF-8, replacing data that is not valid UTF-8 with a ! character.
This behavioral change can be temporarily reverted by setting runtime guard envoy.reloadable_features.service_sanitize_non_utf8_strings to false.
Example exploit or proof-of-concept
Sending a request with non-UTF-8 header values will reproduce this issue when an gRPC upstream is configured.
Mitigation
Setting failure_mode_allow: false for ext_authz.
Discoverer/credits
Marek Szlagor [email protected]
Related news
Red Hat Security Advisory 2023-4623-01 - Red Hat OpenShift Service Mesh is Red Hat's distribution of the Istio service mesh project, tailored for installation into an OpenShift Container Platform installation.
Red Hat OpenShift Service Mesh 2.2.9 Red Hat Product Security has rated this update as having a security impact of Important. A Common Vulnerability Scoring System (CVSS) base score, which gives a detailed severity rating, is available for each vulnerability from the CVE link(s) in the References section.This content is licensed under the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/). If you distribute this content, or a modified version of it, you must provide attribution to Red Hat Inc. and provide a link to the original. Related CVEs: * CVE-2023-27487: A flaw was found in envoy. The header x-envoy-original-path should be an internal header, but Envoy does not remove this header from the request at the beginning of request processing when it is sent from an untrusted client. The faked header could then be used for trace logs and grpc logs, used in the URL for jwt_authn checks if the jwt_authn filter is used, and any other upstr...
gRPC contains a vulnerability that allows hpack table accounting errors could lead to unwanted disconnects between clients and servers in exceptional cases/ Three vectors were found that allow the following DOS attacks: - Unbounded memory buffering in the HPACK parser - Unbounded CPU consumption in the HPACK parser The unbounded CPU consumption is down to a copy that occurred per-input-block in the parser, and because that could be unbounded due to the memory copy bug we end up with an O(n^2) parsing loop, with n selected by the client. The unbounded memory buffering bugs: - The header size limit check was behind the string reading code, so we needed to first buffer up to a 4 gigabyte string before rejecting it as longer than 8 or 16kb. - HPACK varints have an encoding quirk whereby an infinite number of 0’s can be added at the start of an integer. gRPC’s hpack parser needed to read all of them before concluding a parse. - gRPC’s metadata overflow check was performed per frame, so ...