Apache Flink for Streaming — A Field Guide

Flink's defining property versus most stream processors is that it treats streaming as the general case and batch as a special case of it, not the other way around. A Flink job processes an unbounded stream of events with exactly-once state consistency, low latency, and — the part that makes it worth the operational cost — a checkpointing model that survives node failures without losing or duplicating state.

DataStream API versus the Table/SQL API

Flink exposes the same execution engine through two programming models. The DataStream API (Java/Scala/PyFlink) gives full control over state, timers, and windowing for complex event-driven logic. The Table API and Flink SQL let you express the same joins and aggregations declaratively, and increasingly this is the default entry point — Flink SQL over a Kafka source and a JDBC or Iceberg sink covers most ETL-shaped streaming jobs without writing a custom operator.

CREATE TABLE orders (
  order_id STRING,
  amount DECIMAL(10,2),
  order_time TIMESTAMP(3),
  WATERMARK FOR order_time AS order_time - INTERVAL '5' SECOND
) WITH (
  'connector' = 'kafka',
  'topic' = 'orders',
  'properties.bootstrap.servers' = 'kafka:9092',
  'format' = 'json'
);

SELECT
  window_start,
  window_end,
  SUM(amount) AS revenue
FROM TABLE(
  TUMBLE(TABLE orders, DESCRIPTOR(order_time), INTERVAL '1' MINUTE)
)
GROUP BY window_start, window_end;

Event time versus processing time, and watermarks

The single most consequential design decision in a Flink job is whether windows are keyed on event time (when the event actually happened, per its own timestamp) or processing time (when Flink saw it). Event time gives correct, reproducible results even when events arrive late or out of order, at the cost of needing watermarks — a heuristic that tells Flink "I don't expect events older than this to arrive." Get the watermark strategy wrong (too aggressive) and late events get silently dropped; too lax, and windows stay open longer than needed, delaying results.

Checkpointing and exactly-once semantics

Flink's checkpointing (based on the Chandy-Lamport distributed snapshot algorithm) periodically persists each operator's state to durable storage (S3, HDFS, or a filesystem). On failure, Flink restarts from the last completed checkpoint and replays the source from the corresponding offset, giving exactly-once processing guarantees end-to-end when the source (e.g. Kafka) and sink both support it. This is the feature that justifies Flink's operational complexity over a simpler consumer-and-process loop: state and progress are recoverable, not just the fact that a message was consumed.

Choosing a state backend

Flink supports keeping operator state in the JVM heap or in an embedded RocksDB instance on local disk. Heap state is faster but caps out at whatever memory the TaskManager has; RocksDB state spills to disk and scales to state sizes far larger than memory, at the cost of serialization overhead per access. Jobs with large keyed state — deduplication windows over millions of keys, for instance — should default to RocksDB rather than discovering the heap limit in production.

Wrapping up

Flink earns its complexity on jobs where correctness under failure and out-of-order data actually matters — financial aggregation, fraud detection, real-time inventory reconciliation. For simpler fan-out or transform-and-forward pipelines, a lighter tool may be enough; reach for Flink when you need event-time correctness and exactly-once state, not just "process messages fast."