Topics & Partitions
20 min
A Kafka topic is a named stream of messages, but it isn't stored as
one single ordered log — it's split into multiple partitions, each
its own independently-appended, ordered log. A message's key
deterministically decides which partition it lands in (typically hash(key) % num_partitions), which guarantees every message with the same key
always lands in the same partition — and therefore stays in order
relative to every other message with that key.
def deterministic_hash(s):
h = 0
for c in s:
h = (h * 31 + ord(c)) % (2 ** 32)
return h
def assign_partition(key, num_partitions):
return deterministic_hash(key) % num_partitions
This uses a hand-rolled polynomial hash, NOT Python's built-in hash() -- CPython randomizes hash() for strings by default (a security feature called hash randomization), so the same key would map to a DIFFERENT partition on every process restart, breaking the 'same key always goes to the same partition' guarantee entirely.
Using the provided `assign_partition(key, num_partitions)`, write `partition_messages(messages, num_partitions)`: `messages` is a list of `(key, value)` pairs. Route each message to its assigned partition, and return a list of `num_partitions` lists, each holding the `(key, value)` pairs routed to it, IN ORIGINAL ORDER within each partition.
Why does Kafka guarantee message ORDER only within a single partition, not across an entire topic?
You can route keyed messages to partitions deterministically, and understand why Kafka only guarantees ordering within a partition, not across a whole topic.