Real-Time Data Pipelines — A Field Guide

A real-time pipeline is not a batch pipeline running more often — it's a different set of guarantees. Batch ETL gets to assume the data is complete before it processes anything; a streaming pipeline has to produce correct-enough answers continuously while data is still arriving, some of it late, some of it out of order, some of it duplicated because a producer retried. Most of the actual engineering in "real-time data" is in handling those three problems, not in the transport layer that moves bytes around.

The log is the source of truth

The pattern underneath almost every modern streaming architecture is the append-only log — Kafka, Redpanda, Kinesis, Pulsar all implement some variant of it. Producers append events; consumers read from an offset and track their own position. This decouples producers from consumers completely: a consumer that's down for an hour doesn't lose data, it just resumes from where it left off, and you can add a new consumer reading the same stream from the beginning without touching the producer. The log is also what makes replay possible — if a downstream bug corrupted three days of aggregates, you fix the bug and reprocess from the retained log instead of trying to patch the damage after the fact.

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    "orders.events",
    bootstrap_servers="kafka:9092",
    group_id="fraud-scoring",
    enable_auto_commit=False,   # commit only after successful processing
    auto_offset_reset="earliest",
)

for msg in consumer:
    try:
        process(msg.value)
        consumer.commit()  # advance offset only on success
    except Exception:
        log.exception("processing failed, will retry on next poll")
        break

At-least-once vs exactly-once

True exactly-once delivery across a network is not achievable in the general case — what streaming frameworks like Kafka Streams or Flink actually give you is exactly-once processing semantics, achieved by making the side effects idempotent (transactional writes, deduplication keys) rather than by magically preventing redelivery. In practice, at-least-once delivery plus an idempotent consumer (upsert on a unique event ID instead of insert) gets you the same correctness with far less operational complexity than a distributed transaction across producer, broker, and consumer. Decide this early — retrofitting idempotency after you've shipped a pipeline that assumes each event arrives exactly once is a much bigger job than designing for duplicates from day one.

Windowing and late data

Streaming aggregation happens over windows — "orders per minute," "rolling 5-minute error rate" — and the hard part isn't computing the window, it's deciding when to close it. Event time (when something actually happened) and processing time (when your pipeline saw it) diverge constantly: a mobile client buffers events offline and replays them an hour later, a retry arrives after a network blip. Frameworks like Flink handle this with watermarks — a heuristic for "how late can data be before we stop waiting and emit the result" — and allow late data to trigger a correction to an already-emitted window rather than silently dropping it. If your pipeline closes windows purely on wall-clock arrival time, you will quietly undercount anything that shows up late, and nothing will alert you to it because there's no error, just a number that's slightly wrong.

CREATE TABLE orders (
  order_id STRING,
  amount   DECIMAL(10,2),
  event_time TIMESTAMP(3),
  WATERMARK FOR event_time AS event_time - INTERVAL '30' SECOND
) WITH (...);

SELECT
  window_start,
  window_end,
  SUM(amount) AS revenue
FROM TABLE(
  TUMBLE(TABLE orders, DESCRIPTOR(event_time), INTERVAL '1' MINUTE)
)
GROUP BY window_start, window_end;

Backpressure and consumer lag

The metric that tells you a real-time pipeline is actually healthy is consumer lag — the gap between the latest offset in the log and the offset your consumer has processed — not throughput. A pipeline can process a million events a minute and still be "real-time" in name only if lag is climbing, because that means the gap between when something happened and when your system reflects it is growing without bound. Lag climbs for boring reasons: a downstream database that's slower than the ingest rate, a consumer doing synchronous I/O per event instead of batching, or simply under-provisioned consumer parallelism relative to partition count. Alert on lag trend, not absolute throughput.

Wrapping up

Real-time pipelines earn their complexity when the business genuinely needs sub-minute freshness — fraud scoring, live inventory, operational alerting. The core engineering problems are the same regardless of which broker or framework you pick: treat the log as the durable source of truth so consumers can fail and resume safely, make processing idempotent instead of chasing exactly-once delivery, handle late-arriving data with watermarks rather than pretending events always arrive in order, and watch consumer lag as your primary health signal. Get those four right and the specific technology choice becomes a much smaller decision.