Observability for Data Pipelines

Application observability tools — traces, request latency, error rates — answer "is the service up and responding correctly." Data pipelines fail differently: the DAG can run green, every task can exit with status success, and the data at the end can still be wrong, stale, or missing rows, because a pipeline's correctness depends on the data flowing through it, not just the code executing. Observability for data pipelines has to answer a different question: not "did it run," but "should I trust what it produced."

Data freshness and lag monitoring

Freshness is the gap between when data should have landed and when it actually did. A dashboard powered by an hourly pipeline that silently stopped running six hours ago looks identical to one that's current, unless you're explicitly checking the timestamp of the most recent successful load against the expected cadence. This is usually the first and cheapest check to add, and it catches an entire class of failure — a stuck upstream source, a paused Airflow DAG, a Kafka consumer that's fallen behind — that a purely task-level "did it succeed" check misses entirely.

SELECT
  table_name,
  MAX(loaded_at) AS last_load,
  EXTRACT(EPOCH FROM (NOW() - MAX(loaded_at))) / 3600 AS hours_stale
FROM pipeline_metadata.load_log
GROUP BY table_name
HAVING NOW() - MAX(loaded_at) > INTERVAL '2 hours';  -- alert threshold per table

Row-count and schema drift checks

Row counts moving sharply outside their normal range is one of the highest signal-to-noise checks available — a table that usually gets 50,000 new rows a day landing with 400 rows, or with 4 million, is almost always a bug somewhere upstream, whether that's a broken filter, a partial extract, or a duplicated join. Schema drift is the quieter cousin: a source system renames a column, changes a type from integer to string, or starts emitting a new nullable field, and downstream transformations either silently drop data or throw an opaque type error three stages later. Comparing the observed schema against an expected schema at ingestion time — before the bad data propagates — turns that into a single clear failure at the point of entry.

Lineage: knowing what depends on what

When a table breaks, the useful question is immediately "what else does this feed?" Without lineage — a graph of which tables, dashboards, and models consume which upstream tables — that answer lives in someone's memory or a stale wiki page. Tools like dbt (via its manifest and the `dbt docs` DAG), OpenLineage, or a catalog like DataHub can generate this automatically from the transformation code itself, so an alert on a broken source table can also tell you which five downstream dashboards are now serving stale or wrong numbers, instead of finding out from a confused stakeholder.

Distinguishing pipeline failure from upstream data quality

These look identical from an alert's perspective but need entirely different responses. A pipeline failure — an OOM'd Spark job, a timeout, a bad deploy — is your bug, fixable by rerunning or rolling back your own code. An upstream data quality issue — a source API silently changing its response format, a partner feed sending duplicate records, a sensor reporting nulls after a firmware update — is not something a pipeline rerun fixes, because the pipeline is faithfully processing bad input. Separating these means checking the pipeline's own execution logs and metrics first (did the job actually error, or exit cleanly with bad data) before assuming a rerun will help — rerunning against unchanged bad upstream data just reproduces the same bad output.

Wrapping up

Observability for data pipelines means watching the data, not just the job status: freshness against expected cadence, row counts against historical norms, schema against what downstream consumers expect, and lineage so a break's blast radius is visible immediately. A pipeline that finishes green with garbage inside it has failed just as badly as one that crashed — it's just failed more quietly.