Delivery Guarantees
18 min
Kafka's default delivery guarantee is at-least-once: if a consumer crashes after processing a message but before committing its offset, that message gets redelivered on restart — so a consumer must be prepared to see the same message more than once. Making the consumer idempotent (processing the same message twice has the same effect as processing it once) closes this gap: track which message IDs have already been handled, and skip anything already seen.
def dedup_process(messages):
seen = set()
result = []
for msg_id, payload in messages:
if msg_id not in seen:
seen.add(msg_id)
result.append(payload)
return result
This is the exact same 'build a set, check membership' pattern from Data Structures' hash-tables module -- deduplication is fundamentally a set-membership problem, whether you're deduping API requests, DB writes, or Kafka messages.
Write `dedup_process(messages)`: `messages` is a list of `(message_id, payload)` pairs, possibly containing REDELIVERED duplicates (same `message_id` appearing more than once). Return a list of `payload`s, processing each unique `message_id` only once, in first-seen order.
Kafka's default guarantee is AT-LEAST-ONCE delivery (a message might be redelivered after a consumer crash before it commits its offset). How does an idempotent consumer turn that into effectively-exactly-once PROCESSING?
You can implement idempotent message processing via ID-based deduplication, turning Kafka's at-least-once delivery into effectively-once processing.