Apache Pulsar Streaming — A Field Guide

Pulsar's pitch against Kafka comes down to one architectural decision: it separates the serving layer (brokers) from the storage layer (Apache BookKeeper). Brokers are stateless and handle routing and protocol; BookKeeper bookies own the durable log storage. That split is what enables Pulsar's most distinctive operational property — brokers can be added, removed, or rebalanced without moving any data, because the data never lived on the broker in the first place.

Segmented storage versus Kafka's partition model

Kafka ties a partition's data to the broker that owns it, so rebalancing partitions across brokers means physically moving log segments. Pulsar topics are split into ledgers stored across a pool of bookies, striped and replicated independently of any single broker. The practical effect: scaling brokers up or down, or recovering from a broker failure, doesn't trigger a data-copying rebalance — the new broker just starts serving the same ledgers from BookKeeper.

Four subscription models, not one consumer group shape

Kafka has one consumption model: consumer groups, where each partition is read by exactly one consumer in the group. Pulsar supports four subscription types on the same topic simultaneously — exclusive (one consumer), failover (one active, others standby), shared (round-robin across consumers, like a work queue), and key_shared (messages with the same key always go to the same consumer, giving ordering per key while still load-balancing across consumers). Key_shared is the one with no direct Kafka equivalent and is genuinely useful for per-entity ordering (e.g., all events for one order ID processed in order) without needing one partition per entity.

Consumer consumer = client.newConsumer()
    .topic("persistent://tenant/ns/orders")
    .subscriptionName("order-processors")
    .subscriptionType(SubscriptionType.Key_Shared)
    .subscribe();
// messages with the same orderId key always land on
// the same consumer instance, preserving per-key order
// while still spreading load across the subscription

Tiered storage for cheap long-term retention

Because BookKeeper already treats a topic as a sequence of immutable segments, Pulsar can offload older segments to S3 or another object store while keeping recent segments on local disk — transparently, with no change to how consumers read the topic. This gives Kafka-style low-latency reads on hot data and cheap, effectively unlimited retention on cold data without running a separate archival pipeline, which is a real operational simplification versus bolting tiered storage onto Kafka after the fact.

Multi-tenancy as a first-class concept

Pulsar topics are namespaced as tenant/namespace/topic, with quotas, retention, and access policies configurable per tenant and per namespace out of the box. Kafka has no native concept of a tenant — multi-tenant Kafka clusters are usually built with naming conventions and ACLs layered on top by convention, not enforced by the broker. If you're building a platform that serves genuinely separate customers or business units off one cluster, Pulsar's namespace isolation removes a layer of tooling you'd otherwise build yourself.

Wrapping up

Pulsar's architecture is genuinely different from Kafka's, not just a re-skin — the broker/bookie split, native multi-tenancy, and key_shared subscriptions solve real problems Kafka users build workarounds for. That said, Kafka's ecosystem maturity and simpler single-layer operational model still win by default; reach for Pulsar when one of its specific architectural advantages maps directly onto a problem you actually have.