gRPC Load Balancing — A Field Guide

gRPC load balancing breaks the assumption most infrastructure teams carry over from HTTP/1.1: that a connection roughly equals a request, so a plain TCP or L4 load balancer spreading connections across backends spreads load evenly. gRPC's whole performance advantage comes from HTTP/2 multiplexing many RPCs over one long-lived connection — which means an L4 load balancer sees one connection, sends it to one backend, and every RPC on that connection lands on that same backend for the connection's entire lifetime. Load doesn't spread; it piles up.

Why L4 load balancing fails for gRPC

A traditional load balancer operating at the connection level (L4) does its job once, at connection setup, and then gets out of the way — fine for HTTP/1.1 where a busy client opens many short-lived connections, so the balancer gets many chances to spread load. A gRPC client opens a connection and reuses it for potentially thousands of RPCs. Point an L4 balancer at a fleet of gRPC servers and, over time, connections settle unevenly — a handful of backends end up handling most of the traffic while others idle, because the balancer never revisits the decision it made once at connection time.

Two real fixes: client-side and proxy-based

There are two accepted solutions, and they trade off differently. Client-side load balancing has the gRPC client itself resolve a list of backend addresses (via DNS, or a service discovery system like Consul or Kubernetes headless services) and pick which backend to send each RPC to, spreading RPCs across many connections it maintains directly. This removes the load balancer as a bottleneck entirely but pushes balancing logic and backend-health awareness into every client. Proxy-based (L7) load balancing puts an HTTP/2-aware proxy — Envoy, Linkerd, or a cloud load balancer with gRPC support — in front of the backends; the proxy terminates the client connection and load-balances individual streams/RPCs across backend connections it manages, which is transparent to clients but adds a hop and requires the proxy to genuinely understand HTTP/2 framing, not just TCP.

{
  "loadBalancingConfig": [{ "round_robin": {} }],
  "methodConfig": [{
    "name": [{ "service": "orders.v1.OrderService" }],
    "retryPolicy": {
      "maxAttempts": 3,
      "initialBackoff": "0.1s",
      "maxBackoff": "1s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }]
}

The Kubernetes trap: Services default to L4

A default Kubernetes Service (ClusterIP) load-balances at the connection/L4 level via kube-proxy, which reproduces exactly the problem described above for gRPC traffic between pods. The common fixes are either a headless Service combined with client-side load balancing in the gRPC client (so the client resolves individual pod IPs via DNS and balances itself), or routing gRPC traffic through a service mesh sidecar (Linkerd, Istio) or an ingress controller that explicitly understands HTTP/2 and gRPC (a plain nginx L4 passthrough does not).

Health-aware balancing matters more, not less

Because a gRPC connection is long-lived, a backend that becomes unhealthy after a client has already connected keeps receiving RPCs on that connection until something actively detects the failure and reroutes — there's no natural connection churn to self-heal the way short HTTP/1.1 connections provide. gRPC's health checking protocol (grpc.health.v1.Health) exists specifically so load balancers and clients can actively probe backend health and pull unhealthy backends out of rotation quickly, rather than relying on a request eventually failing.

Wrapping up

Default L4 load balancing silently breaks under gRPC's connection-reuse model — it looks fine in a load test with few clients and fails visibly in production with many. Fix it with either client-side balancing against resolved backend addresses or an HTTP/2-aware L7 proxy, and pair either approach with active gRPC health checking so unhealthy backends actually get removed from rotation instead of quietly eating a share of every client's long-lived connection.