Partitioning
20 min
Once a dataset is split into partitions (chunks that can live on
different machines), Spark processes each partition independently and
IN PARALLEL — this is the same "split into pieces, compute in parallel
on each piece" idea from Parallel Computing's parallel-algorithms
module. An aggregate like sum() uses the classic map-reduce shape:
compute a partial sum per partition (the parallel part), then combine
those few partial sums into one final total (the fast, sequential
"reduce" part).
def partition_data(data, num_partitions):
partitions = [[] for _ in range(num_partitions)]
for i, item in enumerate(data):
partitions[i % num_partitions].append(item)
return partitions
def partitioned_sum(data, num_partitions):
partitions = partition_data(data, num_partitions)
partial_sums = [sum(p) for p in partitions]
return sum(partial_sums)
Notice the final combine step only has to add together num_partitions small numbers (here, just 4), not scan all 100 original values again -- this is exactly why map-reduce-shaped aggregation scales well: the expensive full-dataset scan happens in parallel, and the sequential 'stitch it together' step stays cheap regardless of how large the original dataset was.
Write `partitioned_sum(data, num_partitions)`: split `data` round-robin into `num_partitions` groups (like Kafka's partition assignment), compute each partition's sum independently, then combine (reduce) those partial sums into one final total.
Why does Spark split a dataset into partitions and compute a result like a sum in two stages (per-partition sums, then combine) instead of just summing the whole dataset in one pass?
You can implement partition-and-reduce aggregation, the map-reduce shape that lets Spark (and distributed computing generally) scan huge datasets in parallel while keeping the final combine step cheap.