Producers & Consumers
18 min
A Kafka partition is an append-only log, and each message in it has a sequential position called an offset. Consumers don't get messages pushed to them automatically — they PULL, tracking their own position by periodically committing the offset of the last message they've fully processed. This is what makes consumption resumable: after a crash or restart, a consumer just reads its last committed offset and continues from exactly there, rather than losing its place.
def consume_from_offset(log, committed_offset, batch_size):
batch = log[committed_offset:committed_offset + batch_size]
new_offset = committed_offset + len(batch)
return batch, new_offset
Notice consume_from_offset never mutates the log itself -- reading is non-destructive in Kafka (unlike, say, popping a queue), which is exactly why MULTIPLE independent consumer groups can each read the same topic at their own pace without interfering with each other.
Write `consume_from_offset(log, committed_offset, batch_size)`: return a tuple `(batch, new_offset)` where `batch` is up to `batch_size` messages from `log` starting at `committed_offset`, and `new_offset` is `committed_offset` advanced by however many messages were actually read.
Why does a Kafka consumer track a COMMITTED OFFSET instead of just remembering 'the last message I saw' in memory?
You can implement offset-based consumption that resumes correctly after a restart, the mechanism that makes Kafka consumers durable and independently pace-able.