"Exactly-once" is one of the most overloaded terms in distributed systems, and Kafka's marketing hasn't always helped: at the network level, a message can still be sent twice if a producer retries after an ack is lost. What Kafka actually guarantees, when configured correctly, is exactly-once processing semantics for the read-process-write pattern — consume from a topic, do some work, produce to another topic — not a magic guarantee that spans arbitrary external side effects.
Idempotent producers solve duplicate sends
The most common source of duplicates pre-idempotence was retries: a producer sends a message, the broker writes it and acks, the ack is lost in the network, the producer times out and retries, and now the message is written twice. Setting enable.idempotence=true (default since Kafka 3.0 when acks=all) fixes this at the producer level: each producer gets a unique producer ID, and every message carries a sequence number per partition. The broker tracks the last sequence number it wrote for each producer/partition pair and rejects a duplicate with a matching sequence number instead of writing it again — so retries are safe without any application-level dedup logic.
enable.idempotence=true
acks=all
max.in.flight.requests.per.connection=5
retries=2147483647
delivery.timeout.ms=120000An idempotent producer guarantees no duplicates within a single partition. It does not guarantee that a batch of writes across multiple partitions — or a read-then-write spanning a consumer offset commit and a producer send — happens atomically as a single unit. That's what transactions add on top.
Transactions make multi-partition writes atomic
Kafka transactions let a producer write to multiple partitions (and topics) and have those writes become visible to consumers all at once or not at all, using a transaction coordinator and a two-phase-commit-like protocol. This is what makes the "consume, transform, produce" pattern exactly-once end to end: the consumer's offset commit (marking the input message as processed) and the producer's output writes are wrapped in the same transaction, so a crash between processing and committing the offset doesn't result in either a lost update or a duplicate reprocessing on restart.
producer.initTransactions();
try {
producer.beginTransaction();
ConsumerRecords records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord record : records) {
String transformed = transform(record.value());
producer.send(new ProducerRecord<>("output-topic", record.key(), transformed));
}
producer.sendOffsetsToTransaction(
currentOffsets(records), consumer.groupMetadata());
producer.commitTransaction();
} catch (Exception e) {
producer.abortTransaction();
} Consumers need read_committed to see the guarantee
Transactional writes are only exactly-once from a consumer's perspective if that consumer sets isolation.level=read_committed. With the default read_uncommitted, a consumer sees every message written to a partition, including ones from transactions that were later aborted — which defeats the purpose. Every downstream consumer in an exactly-once pipeline needs this setting; missing it on even one hop reintroduces exactly the duplicate/inconsistent reads the transactional producer was meant to prevent.
Where the guarantee stops: external side effects
Kafka's transactions cover Kafka-to-Kafka writes. The moment your consumer's processing has a side effect outside Kafka — an HTTP call to another service, a write to a non-transactional external database, sending an email — Kafka's exactly-once guarantee doesn't extend to that side effect. A crash after the external call succeeds but before the Kafka offset commits will cause a reprocessing that repeats the external call. For those cases you still need the external system to be idempotent (an idempotency key on the API call, an upsert instead of an insert) or you need that external write to participate in a genuinely distributed transaction, which Kafka alone can't give you.
Setting processing.guarantee=exactly_once_v2 in a Kafka Streams application wires up the idempotent producer, transactions, and read_committed consumption automatically for you across the whole topology. It's the easiest way to get this right if your pipeline is expressible as a Streams application rather than raw producer/consumer code.
Wrapping up
Kafka's exactly-once semantics are real but scoped: idempotent producers stop duplicate sends from retries, transactions make multi-partition read-process-write atomic, and read_committed consumers are required to actually observe that atomicity. None of it extends automatically to side effects outside Kafka — those still need their own idempotency strategy. Understanding that boundary is most of what separates "we turned on exactly-once and it still duplicated things" from a pipeline that actually behaves the way the name promises.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.