Grafana Tempo Distributed Tracing

Every other piece of the observability stack scales by indexing everything up front — metrics get labels, logs get streams, and both get expensive the moment you index a high-cardinality field. Traces broke that model for years, because the whole point of a trace is a request ID nobody predicted you'd search for. Grafana Tempo's answer is to stop indexing traces at all: store them as cheap blobs in object storage and rely on TraceQL plus trace IDs handed to you by metrics and logs to find the one you need. It's a trade that only works because the rest of the LGTM stack (Loki, Grafana, Tempo, Mimir/Prometheus) is designed to hand you that ID in the first place.

Why Tempo skips the index

Jaeger and Zipkin both learned the hard way that indexing every span attribute for full-text search gets expensive fast — Elasticsearch or Cassandra backing a trace store means you're paying for a search index sized to your trace volume, not your query volume. Tempo's design bet is that you almost always arrive at a trace with its ID already in hand: a user pastes it from an error page, an exemplar links it from a Prometheus graph, a log line in Loki has it as a field. So Tempo only really needs one index — trace ID to storage location — and everything else can live as flat, compressed blocks in S3, GCS, Azure Blob, or even local disk. That one design decision is why Tempo can be an order of magnitude cheaper to run than a search-indexed trace store at the same retention.

Ingesting traces with OTLP

Tempo speaks OTLP natively, which in practice means your services don't send traces to Tempo directly — they send them to an OpenTelemetry Collector, which batches, maybe samples, and forwards to Tempo's OTLP endpoint. This decoupling matters operationally: the collector can retry on Tempo restarts, apply tail-based sampling before anything hits storage, and fan out the same trace data to more than one backend if you're migrating off Jaeger.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  otlp/tempo:
    endpoint: tempo.observability.svc:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

Tempo's own config just needs to know where to store the blocks. A minimal S3-backed setup looks like this — the important part is that Tempo treats the object store as the source of truth, with only a small amount of local disk used for the write-ahead log before a block gets flushed and compacted.

storage:
  trace:
    backend: s3
    s3:
      bucket: prod-tempo-traces
      endpoint: s3.amazonaws.com
      region: eu-west-1
    wal:
      path: /var/tempo/wal
    block:
      version: vParquet4

compactor:
  compaction:
    block_retention: 336h

Querying with TraceQL

TraceQL is Tempo's query language, and it reads like a cross between PromQL's filtering and a span-shaped selector. Instead of full-text search across span attributes, you write structured queries that match on span or resource attributes, durations, and status codes, and TraceQL walks the block index to find matching traces without ever building a general-purpose inverted index. A query to find slow, failing calls to a payments service looks like this:

{ span.service.name = "payments-api"
  && span.http.status_code >= 500
  && duration > 800ms }
  | select(span.http.route, span.http.status_code)

What makes this different from grepping logs is that the query is span-shaped rather than line-shaped: you can filter on a parent span's attributes while selecting children, or ask for traces where any span in the whole tree matches a condition, which is exactly the kind of question a log line can't answer because it has no notion of "this happened inside that other thing."

Correlating with metrics and logs via exemplars

The part of the LGTM story that actually changes how you debug day to day isn't Tempo in isolation — it's exemplars. When your app is instrumented with OpenTelemetry and scraped by Prometheus (or remote-written into Mimir), each metric sample can carry an exemplar: a trace ID captured at the moment that data point was recorded. In Grafana, a latency histogram panel backed by Mimir shows exemplar dots on the graph, and clicking one takes you straight to the Tempo trace for that exact slow request — no copy-pasting IDs between tabs.

Loki closes the other side of the loop. If your log lines include the trace ID as a structured field (which most OpenTelemetry logging bridges do automatically), Grafana's Tempo data source can derive a link from a log line straight to the matching trace, and from a trace's spans back to the logs emitted during that span. In practice this means an on-call engineer starts on a dashboard, jumps to the exact trace behind a latency spike, and from there jumps to the exact log lines from the service that misbehaved — three tools, one investigation, no manual correlation.

Operating Tempo in production

Tempo's operational profile is unusually calm once it's wired up correctly, mostly because there's no query-time indexing work happening on the write path — ingestion is mostly "accept spans, batch them, flush a block to object storage." The places it actually needs attention are compaction (make sure the compactor keeps up, or you end up with thousands of small blocks and slow queries), sampling (100% trace retention is rarely necessary or affordable at real traffic volumes — head-based sampling in the collector or tail-based sampling for "keep all the errors and slow ones" is the usual compromise), and making sure your object storage bucket has sane lifecycle rules so the compactor's `block_retention` setting isn't fighting a separate S3 lifecycle policy that deletes blocks out from under it.

Wrapping up

Tempo's real contribution isn't a faster trace store — it's proving that traces don't need to be indexed like logs to be useful, because in practice you arrive at a trace with its ID already in hand from a metric or a log line. That single design choice is why it can sit on plain object storage and stay cheap at retention levels that would be painful with a search-indexed backend. The catch is that the payoff depends entirely on whether your metrics and logs are actually propagating trace IDs to begin with — Tempo by itself is just cheap storage with a decent query language; the LGTM story only clicks once exemplars and log correlation are wired through your whole stack.