gRPC Tracing — A Field Guide

A single user action in a microservices architecture can fan out across a dozen gRPC calls before it produces a response, and when one of those calls is slow, logs on each service in isolation rarely show you which hop is the actual culprit. Distributed tracing solves this by propagating a trace context across every RPC boundary so all the spans for one logical request — across every service it touched — can be reassembled into one timeline. For gRPC specifically, this is largely a solved problem: OpenTelemetry ships interceptors purpose-built for it.

Interceptors are where tracing hooks into gRPC

gRPC's interceptor mechanism — middleware that wraps every unary or streaming call on the client or server side — is exactly the hook OpenTelemetry uses to instrument gRPC without touching business logic. The OpenTelemetry gRPC instrumentation library provides a client interceptor that injects the current trace context into outgoing call metadata, and a server interceptor that extracts it from incoming metadata and starts a child span. Wire both in once, at server and client construction time, and every RPC in the service is traced automatically — no per-method instrumentation required.

import (
    "google.golang.org/grpc"
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)

server := grpc.NewServer(
    grpc.StatsHandler(otelgrpc.NewServerHandler()),
)

conn, err := grpc.Dial(target,
    grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
)

Trace context travels in gRPC metadata

A trace needs a way to cross the network boundary between services, and gRPC provides exactly that mechanism: metadata, the gRPC equivalent of HTTP headers. The W3C Trace Context standard (traceparent header) defines the format, and the OpenTelemetry propagator serializes the current trace ID, span ID, and sampling decision into gRPC metadata on the outgoing call, then deserializes it on the receiving end to continue the same trace rather than starting a new, disconnected one. This is what makes a single trace span five services instead of producing five separate, unlinked traces.

What belongs on a gRPC span

OpenTelemetry's semantic conventions for gRPC define a standard set of span attributes — the full method name (/orders.v1.OrderService/GetOrder), the gRPC status code the call completed with, and message size where relevant — so that traces from different services and different teams look consistent in whatever backend collects them (Jaeger, Tempo, Honeycomb, a cloud vendor's APM). Following the standard conventions instead of inventing custom attribute names is what makes traces queryable and comparable across an organization's services rather than being a one-off per team.

Streaming RPCs need span events, not just start/end timing

A unary call maps cleanly to one span with a start and end time. A streaming RPC that stays open for minutes and sends hundreds of messages needs more granularity than "the stream started, the stream ended" — recording individual messages as span events (with a sequence number or timestamp) lets you see gaps, bursts, or a stall partway through a long-lived stream, which a single start/end span would completely hide.

Sample deliberately — tracing every call adds up

Tracing 100% of RPCs in a high-throughput internal service generates enormous span volume for marginal debugging value once the system is healthy. Head-based sampling (deciding to trace a request at its start, propagated to every downstream span) or tail-based sampling (deciding after the fact, keeping traces that were slow or errored) both reduce volume while preserving the traces you actually care about — the slow ones and the failed ones — rather than tracing everything and drowning the useful signal in routine, fast, successful calls.

Wrapping up

gRPC's interceptor model makes distributed tracing close to a drop-in addition rather than a rewrite: wire OpenTelemetry's client and server interceptors at construction time, propagate trace context through gRPC metadata, and follow the standard semantic conventions for status codes and method names so traces stay queryable across services. Sample deliberately once volume grows, and give streaming RPCs span events, not just a single start/end timer.