Prometheus + Grafana Monitoring — A Field Guide

Prometheus and Grafana get bundled together so often that people assume they're one product. They're not: Prometheus is a time-series database that scrapes and stores metrics and can evaluate alerting rules against them, and Grafana is a visualization layer that can query Prometheus (among many other data sources) and turn the numbers into dashboards. Understanding that split is most of what you need to reason about the stack — the rest is knowing how Prometheus collects data and what PromQL is actually computing when you ask it a question.

The pull-based scraping model

Prometheus is pull-based: your services expose a /metrics HTTP endpoint returning plain text, and Prometheus scrapes that endpoint on a fixed interval — 15 or 30 seconds is typical. This is the opposite of push-based systems like StatsD, where the application actively sends metrics to a collector. Pull-based has a couple of practical advantages: Prometheus knows immediately if a target stops responding (a scrape failure is itself a signal), and instrumenting a service is as simple as exposing an endpoint — no client-side batching or network reliability logic needed in the app. The trade-off is that Prometheus needs to know where your targets are, either via static config or service discovery (Kubernetes, Consul, EC2 tags, etc.).

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'api-service'
    static_configs:
      - targets: ['api-1:9100', 'api-2:9100']
    metrics_path: /metrics

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['api-1:9100', 'api-2:9100', 'worker-1:9100']

rule_files:
  - 'alerts.yml'

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

The four metric types

Prometheus has exactly four metric types, and picking the right one matters because it determines what queries make sense later. A counter only goes up (or resets to zero on restart) — total requests served, total errors, total bytes sent. A gauge can go up or down — current memory usage, queue depth, active connections. A histogram samples observations into configurable buckets and gives you counts, sums, and the ability to estimate quantiles — request latency is the classic case. A summary is similar to a histogram but calculates quantiles client-side, which is cheaper to query but can't be aggregated across instances the way histogram buckets can. In practice: use counters for anything cumulative, gauges for anything that fluctuates, and histograms over summaries unless you have a specific reason not to, since histograms aggregate correctly across multiple replicas.

PromQL basics: rate() over raw counters

The single most common PromQL mistake is graphing a raw counter directly. A counter only increases, so a graph of http_requests_total is just a line going up and to the right forever — it tells you nothing about current traffic. What you actually want is the rate of change, which is what rate() computes: the per-second average rate of increase over a time window, correctly handling counter resets (like a process restart).

# Per-second request rate, averaged over the last 5 minutes
rate(http_requests_total[5m])

# Same, broken down by status code, for error-rate dashboards
sum by (status) (rate(http_requests_total[5m]))

# 95th percentile request latency from a histogram
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

The [5m] range vector says "look at the last 5 minutes of samples for this series," and rate() turns that into a per-second number. Longer windows smooth out spikes and are more forgiving of scrape gaps; shorter windows react faster but get noisier. Five minutes is a reasonable default for most dashboards and alerts.

Grafana and Alertmanager: the two layers on top

Grafana doesn't store any data itself — it queries Prometheus (or Loki, InfluxDB, Postgres, whatever you've configured as a data source) and renders the result as a panel. This means Grafana dashboards are only as good as the PromQL behind them; a beautifully designed dashboard built on a raw-counter query is still useless. For alerting, you have two reasonable paths: Prometheus's own alerting rules, evaluated against the same PromQL and sent to Alertmanager for routing, deduplication, and silencing; or Grafana's built-in alerting, which can also evaluate queries and fire notifications. Alertmanager is the more mature choice if you're already Prometheus-native and want proper grouping and inhibition rules; Grafana alerting is convenient if you want everything — dashboards and alerts — configured in one UI.

The cardinality trap

The operational gotcha that catches nearly everyone eventually is cardinality. Every unique combination of label values on a metric creates a new time series, and Prometheus keeps each series in memory. A metric like http_requests_total{status="200", route="/orders"} is fine — a handful of routes times a handful of status codes is a small, bounded set. The moment someone adds a label with an unbounded set of values — a user_id, a raw URL with query parameters, a UUID — the number of series explodes, and Prometheus's memory usage climbs until the process falls over or scrapes start timing out.

Wrapping up

Prometheus and Grafana solve different halves of the same problem: Prometheus pulls metrics from your services on a schedule, stores them as time series, and can evaluate alerting rules against them; Grafana turns whatever's stored there into dashboards, usually with PromQL queries that use rate() on counters rather than graphing raw values. Get the metric types right, keep label values bounded, and decide up front whether Alertmanager or Grafana owns your alerting — the stack is straightforward once those four things are settled.