A failed payment request means a Stripe user did all the work to get a customer to pay, but lost the sale because of us. We never want this to happen. To minimize failed payments for our users, we target 99.9995% reliability. Maintaining that level of reliability while processing around 1.6% of global GDP is an exceptionally difficult technical challenge, but most of us who join Stripe’s infrastructure teams are here because we love technical challenges.
For almost 10 years, we used Envoy as our proxy for Stripe’s service mesh. It worked well for us for years, but as Stripe’s traffic grew and we increased our reliability targets, we started running into critical issues that made us wonder if we needed our own proxy.
We decided to try building a replacement for Envoy. We wanted better reliability and scalability, but we weren’t sure if we’d be able to match Envoy’s latency or CPU footprint. This was a technical risk: if our solution ended up slower or less efficient than Envoy, it would prevent us from deploying it.
Why did we build it ourselves?
Internal services at Stripe use our service mesh to communicate with each other. The data plane of the service mesh is responsible for request routing, load balancing, authorization, encryption, and other similar features.
Stripe’s service mesh handles millions of requests per second between thousands of services. One public API request relies on many internal service requests, so small improvements in data-plane latency or reliability make a huge difference in maintaining 99.9995% public API uptime.
Over time, a few reliability and latency issues we had with Envoy became large problems we could no longer ignore. For example, under consistent traffic, Envoy’s worker-based threading model means that each worker will usually hold a connection to every upstream endpoint it can route traffic to. The default deployment strategy is to run Envoy with one worker per CPU core, which means each server machine ends up with one outbound connection per client core.
Here is an example of what this would look like with two 8-core client machines load balancing sustained traffic between two server machines:
Every worker on both client machines maintains a connection to both server machines, which means each server machine needs to handle 16 connections for just these two clients. This becomes especially problematic if you run an infrastructure service that handles requests from the entire fleet (e.g., a service that processes application tracing data). That infrastructure service is forced to maintain connections for roughly every CPU core fleetwide, which is many hundreds of thousands of connections. Scaling up a machine enough to handle this many connections is expensive, and often the problem went unnoticed until too many open connections overloaded the service and caused an incident.
Over time, our Envoy rollouts also started to get increasingly risky, which forced us to roll out upgrades more slowly. Hyrum's Law is especially applicable at Stripe because our fleet contains many unique services. This means small behavior changes in Envoy updates would occasionally cause problems for us.
For example, in 2024, we ran into an especially scary issue when trying to roll out an Envoy upgrade. This upgrade pulled in a new version of nghttp2, which introduced a rapid-reset HTTP/2 DDoS defense to protect against a vulnerability. Unfortunately, when using the default settings, this mitigation accidentally rejected some legitimate traffic to one of our services. Luckily, we caught this in a QA load test and fixed the configuration before rolling out to production, but this was a close call. If our load test didn’t replicate the specific traffic pattern the service receives in production, we might not have noticed this regression until it started rejecting real requests. A single production incident like this could jeopardize our 99.9995% reliability goal for the whole year.
Running Envoy also limited our ability to provide custom features for Stripe. For example, Stripe’s Ruby services require big, expensive machines to run. We had designed a new routing approach we could use to distribute traffic more efficiently across backend machines to reduce our Ruby machine sizes and make our Ruby backends more reliable. But this would have required too much customization of Envoy, so we never pursued it.
What we built
Mesh-proxy is our Go proxy and Envoy replacement. To efficiently implement all the features of a proxy, mesh-proxy has its own implementation of HTTP/1.1 and HTTP/2. Mesh-proxy connects to our service registry directly for service discovery and supports retries, hedging, rate limiting, and dynamic priority-based request routing.
Go has a few features that helped make mesh-proxy successful. Goroutines are cheap to spin up, which allows us to spin up new ones for every connection. Go’s official HTTP/2 framing library was very useful, and Go is well-supported at Stripe, so we benefit from great developer tooling.
Our biggest risk in building mesh-proxy was that we weren’t sure if we’d be able to match Envoy’s latency or CPU footprint, but our early load tests looked good compared to our Envoy deployment. After rolling out mesh-proxy, we saw around half the CPU under high load, and roughly a 50% latency reduction compared to our Envoy deployment. At our scale, this saves around 3,000 days of cumulative inter-service latency each day.
To address the connection cardinality problem, we built mesh-proxy with a different concurrency model. Instead of a worker-based model with one worker per core, mesh-proxy uses one read goroutine and one write goroutine per connection. Because mesh-proxy multiplexes requests across a shared connection rather than partitioning requests across many workers, we only need one connection per host, compared to one connection per worker with Envoy.
Here is what the 8-core client machines from above look like after migrating to mesh-proxy:
Each client machine only needs one connection to each server machine, which means the server machines only need to maintain two connections for these two clients. This connection reduction means fleetwide services don’t need to maintain nearly as many connections, which allows them to run reliably on smaller machines. The downside is that this approach required more work to coordinate shared request state across goroutines, but we felt this tradeoff was worth it.
We also found that connecting to our service registry directly from mesh-proxy was more efficient than xDS in many cases. Our Envoy deployment tended to cause more xDS updates than were necessary, which used extra CPU and memory on every machine. Here are a couple of graphs of the most significant example of a service that previously received an enormous number of xDS updates. You can tell when mesh-proxy was rolled out by the massive drop in CPU and memory.
In addition to the performance improvements, rolling out mesh-proxy enabled us to make improvements that were not previously feasible for us with Envoy. We rolled out a feature that uses custom Ruby infrastructure code to balance traffic more evenly between Ruby backend replicas, which gives us 15–20% more headroom before traffic bursts overwhelm the machines. We were also able to roll out an overhaul of our priority-based routing system that has enabled improvements to our deployment infrastructure, rate limiting system, and traffic shifting logic. These features and others on our roadmap were far too custom for us to realistically build into our old Envoy deployment.
What’s next
It’s a little counterintuitive that building something in-house would be more reliable than operating a battle-tested open-source project. But our internal service mesh has become complex and custom enough that we’ve had great success building our own proxy. All mesh-proxy code is reviewed by our team and rigorously tested with unit tests, end-to-end tests, and synthetic tests before deployment.
Because our team operates components that handle communication between almost every service at Stripe, we have a unique advantage for identifying inefficiencies and finding solutions. We’re currently working on capabilities like anomaly detection and smart routing so we can find and fix issues before users feel any impact.
If you’re interested in improving infrastructure at Stripe, we’re hiring.