Kafka Streams vs Kafka — A Field Guide

Half the confusion around Kafka comes from the name being overloaded. "Kafka" is the broker — a durable, partitioned, replayable log that stores records and serves them to consumers. "Kafka Streams" is something else entirely: a Java/Scala client library that you compile into your own application to transform, join, and aggregate those records. You do not deploy a Kafka Streams cluster; you deploy your app, and the library does the stream processing inside it. Getting that distinction straight is what stops teams from over-building.

What Kafka is, and what it is not

The broker's whole job is to accept writes to topics, persist them across partitions with a configurable replication factor, and let consumers read from any offset they choose. It does not transform data. It does not join two topics for you. It does not de-duplicate. Producers append, consumers read, and the broker guarantees ordering within a partition and durability according to your acks and replication settings. Everything past "store and hand back bytes" is the client's problem — which is exactly the gap Kafka Streams (and ksqlDB, and Flink) exist to fill.

Kafka Streams: a library, not a cluster

Kafka Streams gives you a DSL for building a processing topology — a graph of sources, transformations, and sinks — that reads from input topics and writes to output topics. It handles partition assignment, local state, and fault tolerance by piggybacking on Kafka's own consumer group protocol. A minimal topology that filters and re-keys orders looks like this:

StreamsBuilder builder = new StreamsBuilder();
KStream<String, Order> orders = builder.stream(
    "orders", Consumed.with(Serdes.String(), orderSerde));

orders
    .filter((key, order) -> order.total() > 0)
    .selectKey((key, order) -> order.customerId())
    .to("orders-by-customer", Produced.with(Serdes.String(), orderSerde));

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));

Scaling is the elegant part: run the same JAR on three machines with the same application.id and Kafka Streams rebalances partitions across the instances automatically, the same way any consumer group does. There is no separate job manager or resource cluster to operate — the "cluster" is just your app, running more than once.

When a plain consumer is enough

Reach for Kafka Streams when you genuinely need stateful stream processing — windowed aggregations, stream-stream or stream-table joins, or exactly-once transforms. If all you are doing is reading each record and doing something with it (write to a database, call an API, emit a metric), a plain KafkaConsumer is simpler, has fewer moving parts, and is far easier to reason about when it breaks at 2 AM.

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.subscribe(List.of("orders"));
    while (running) {
        ConsumerRecords<String, String> records =
            consumer.poll(Duration.ofMillis(200));
        for (ConsumerRecord<String, String> record : records) {
            handle(record);              // side effect: DB write, HTTP call, etc.
        }
        consumer.commitSync();           // commit only after work succeeds
    }
}

The rule of thumb I use: if you never need to remember anything about previous records to process the current one, you do not need Kafka Streams. Stateless work is a consumer's job.

State stores, changelogs, and exactly-once

The moment your processing becomes stateful — counting events per window, joining a stream against a table — Kafka Streams keeps that state in a local RocksDB store and backs it up to a compacted changelog topic in Kafka. If an instance dies, another instance rebuilds the state by replaying the changelog. This is the feature that makes Kafka Streams worth the added complexity, and also the one that surprises people: your stateful app now depends on extra internal topics that need retention and storage planning.

Kafka Streams vs ksqlDB vs Flink

Kafka Streams is not the only way to process a Kafka topic, and it is not always the best one. The honest comparison:

Wrapping up

Kafka is the log; Kafka Streams is a library that processes the log from inside your application. Start with a plain consumer for anything stateless, adopt Kafka Streams when you need local state and joins without standing up another cluster, and graduate to Flink only when the scale or latency genuinely calls for it. If you are weighing one of these for a real workload, reach out — or keep reading through the rest of the blog.