At Google, we have a commitment to enhancing the security and reliability of the Envoy proxy. We have dedicated initiatives around hardening, fuzzing, and responding to CVEs that are intended to increase our confidence in the trustworthiness of Envoy as a component in our infrastructure stack and those of our customers. In addition, we offer a Vulnerability Reward Program, open to all security researchers who can provide details on security vulnerabilities in Envoy.

Why do we care about making Envoy secure? In addition to being good stewards of OSS projects, we also use Envoy to provide cloud services, like Internal HTTP(S) Load Balancing. For such services, customers depend on Google to uncover and resolve security issues.

This article provides a deep dive into an Envoy vulnerability that was successfully detected and fixed by our Envoy Platform team. The focus is on a single heap vulnerability in Envoy’s HTTP/1 codec which, if exploited, would allow an attacker to bypass Envoy’s access control and routing logic.

Heap vulnerabilities are rare in Envoy as we use modern C++ features such as smart pointers, fuzz the data plane and have high test coverage (97%+ line coverage). We also make use of Clang’s Address Sanitizer, which runs in our CI and during fuzzing. However, even a single heap vulnerability can provide a potent attack vector. We consider one of the contributions of this article to be raising the awareness over how heap vulnerabilities can be exploited on an L7 network proxy’s data plane, from first principles and with relatively little sophistication. This complements another recently published heap vulnerability in HAProxy’s HPACK implementation, CVE-2020–11100 from Google’s Project Zero.

The vulnerability we focus on below is CVE-2019–18801, which was fixed in the 1.12.2 Envoy security release in early December 2019. GCP’s Internal HTTP(S) Load Balancing product (which is based on Envoy), was patched prior to the embargo release date. Envoy-derived binaries, e.g. Istio, were patched in tandem with the release. We were alerted to a potential exploit while investigating a report from one of Envoy’s data plane fuzzers running on ClusterFuzz. It required about 2 days of work to turn into a viable proof-of-concept, which including time learning about tcmalloc internals, experimenting with heap exploits techniques (e.g. heap shaping, vptr manipulation, etc.), building out tooling and working through the story below.

It’s worth noting that the underlying implementation heap overrun bug would have been fixed even if no demonstrated exploit could have been found. Without the demonstration of exploitability, it would have been a correctness bug with potential security implications, and still worth fixing.

When talking about vulnerabilities, it’s helpful to be aware of the threat model. We take Envoy’s documented threat model and consider the case of an untrusted downstream client under the control of an attacker sending HTTP requests to an Envoy proxy in this article.

Read the rest →