OpenTelemetry Instrumentation — A Field Guide

Most teams adopt OpenTelemetry because a vendor's proprietary agent got too expensive, and then treat it as a drop-in replacement — install the SDK, point it at a collector, done. That gets you traces, which is useful, but OpenTelemetry defines three signal types for a reason, and the instrumentation decisions that matter (what to auto-instrument, what to hand-instrument, what to turn into a metric versus a log line) determine whether the system is actually debuggable six months later or just generates a lot of data nobody trusts.

Auto-instrumentation vs manual spans

Auto-instrumentation hooks into known libraries — your HTTP framework, your database driver, your message queue client — and wraps their calls in spans without you writing anything. It's the right starting point for any service: point the OpenTelemetry agent or SDK auto-instrumentation package at your app, and you get a request-in, request-out trace with database and HTTP calls as child spans, for free. Where it falls short is your own business logic — auto-instrumentation can't know that the interesting part of your checkout flow is the fraud-scoring call sandwiched between two database writes. That requires manual spans: wrapping the specific code path you actually want visibility into with the SDK's tracer API, naming it meaningfully, and attaching attributes (order ID, customer tier, whatever you'll want to filter by later).

const tracer = trace.getTracer('checkout-service');

async function scoreForFraud(order) {
  return tracer.startActiveSpan('fraud.score', async (span) => {
    span.setAttribute('order.id', order.id);
    span.setAttribute('order.total_cents', order.totalCents);
    try {
      const result = await fraudClient.score(order);
      span.setAttribute('fraud.score', result.score);
      return result;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

Three signals, not one

OpenTelemetry treats traces, metrics, and logs as distinct signal types, each suited to a different question. A trace answers "what happened during this one request, across every service it touched" — good for debugging a specific slow or failed request. A metric answers "what's the aggregate behavior across thousands of requests" — request rate, error rate, p99 latency — good for dashboards and alerting. A log answers "what exactly happened at this one point in time, in detail" — good for the deep dive once a trace or metric has told you where to look. Using logs as your primary debugging signal because traces feel like extra setup is the most common regression teams make; it works until the system has enough services that grepping logs across all of them stops being feasible.

Exporters and the collector

The OpenTelemetry SDK in your app doesn't talk to your observability backend directly in most setups — it exports spans and metrics to the OpenTelemetry Collector, a separate process that batches, filters, and forwards data to one or more backends (Jaeger, Tempo, Prometheus, a vendor's ingest endpoint). The collector is the layer where you do sampling (deciding which traces to keep, since capturing 100% of traffic is often too expensive), attribute scrubbing (stripping anything that shouldn't leave your network), and multi-backend fan-out, without touching application code.

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
  tail_sampling:
    policies:
      - name: errors-always
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: sample-10-percent
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  otlp:
    endpoint: tempo:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp]

The two pitfalls that recur

Over-instrumenting is the first: wrapping every function call in its own span produces traces so deep and noisy that nobody can read them, and the overhead of creating and exporting that many spans is measurable at scale. Instrument at the boundaries that matter — service calls, database queries, external API calls, and the specific business logic you'll actually want to debug — not every internal function.

Wrapping up

OpenTelemetry gives you a vendor-neutral way to produce traces, metrics, and logs, but the instrumentation choices are still yours to get right: auto-instrument the framework boundaries, hand-instrument the business logic you actually care about, propagate context so traces don't fragment at service edges, and keep high-cardinality data out of metric labels. Get those right and the collector layer lets you swap backends without touching application code again.