Every label added to a metric multiplies the number of distinct time series a monitoring system has to store and query. A counter with no labels is one series. Add a `status_code` label with 5 values and a `region` label with 4 values and you have 20 series from one metric definition. Add a `user_id` label and you can go from 20 to millions, because a metric's cardinality is the product of its label cardinalities, and unbounded labels (user IDs, request IDs, raw URLs) turn a small counter into a system-destabilizing one.
What a label should and shouldn't hold
A good metric label is a value from a small, known, roughly-fixed set: HTTP status code, environment, region, service name. A bad metric label is anything with unbounded or near-unbounded cardinality: user ID, session ID, order ID, raw path segments containing an ID. The rule of thumb that holds up in practice is: if you can't enumerate the realistic set of values in advance, it doesn't belong on a metric — it belongs in a log line or a trace span, where high-cardinality fields are the expected, well-supported case.
// Bad: user_id is effectively unbounded cardinality
requestsTotal.WithLabelValues(userID, path).Inc()
// Good: bounded set of labels
requestsTotal.WithLabelValues(statusCode, route, method).Inc()
// user_id, if needed for debugging, belongs in a structured log line,
// correlated by request_id / trace_id, not on the metric itself.How a cardinality explosion actually hurts you
Time series databases (Prometheus, and most of what sits behind Datadog or Grafana Cloud under the hood) allocate memory and index structures per unique series. A cardinality explosion shows up as memory pressure on the metrics backend, slow or timing-out queries against affected metrics, and in hosted products, a real dollar cost since most usage-based pricing bills per active time series. It rarely announces itself clearly — the first symptom is usually "dashboards are slow" or "Prometheus OOMed again," not an obvious error pointing at the offending metric.
Labeling a metric with the raw request path (/users/48213/orders) instead of the route template (/users/:id/orders) turns one logical endpoint into as many series as there are user IDs that have hit it. Always label with the templated route, never the resolved path.
Audit cardinality before it becomes an incident
Prometheus exposes prometheus_tsdb_symbol_table_size_bytes and per-metric series counts you can query directly; most hosted vendors have an equivalent cardinality explorer. Periodically checking which metrics have the highest series counts, before there's an incident, catches an unbounded label early — the fix (drop the label, or move it to logging) is trivial before it ships, and a multi-team migration once a thousand dashboards depend on the bad label.
Consistent naming reduces accidental metric duplication
A second, quieter cardinality problem is teams defining near-duplicate metrics because naming wasn't consistent — `http_requests_total` in one service and `http_request_count` in another, each with slightly different label sets, tracked as unrelated series that can't be aggregated together. A shared naming convention (Prometheus's own metric-naming guidelines are a reasonable default) and a lightweight review step for new metrics catches this before dashboards fragment across near-duplicates.
A one-line comment on a pull request — "this label looks unbounded, does it need to be on the metric or can it move to the log line" — is far cheaper than the cardinality explosion it prevents. Bake the check into review, not into a dashboard nobody looks at until something breaks.
Wrapping up
Cardinality is the resource limit that catches most teams by surprise in metrics systems, because nothing in the API stops you from adding an unbounded label — it just quietly costs memory, query latency, and money until it doesn't anymore. Keep labels to small, enumerable value sets, push anything with real cardinality (user IDs, request IDs) into logs and traces instead, and audit series counts before they become the incident.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.