Batch Pipelines
16 min
ETL (Extract, Transform, Load) pipelines often deal with datasets far too large to hold in memory all at once. Batch processing handles this by working through the data in fixed-size chunks — extract a batch, transform it, load it, then move to the next batch — keeping memory usage bounded no matter how large the total dataset grows.
def process_in_batches(items, batch_size, transform):
results = []
for i in range(0, len(items), batch_size):
batch = items[i:i + batch_size]
results.extend(transform(batch))
return results
range(0, len(items), batch_size) is the standard Python idiom for chunking a sequence -- it generates the START index of each batch (0, 2, 4, ...), and items[i:i+batch_size] naturally clips the final batch short if fewer than batch_size items remain, exactly like Kafka's consume_from_offset did.
Write `process_in_batches(items, batch_size, transform)`: split `items` into consecutive chunks of at most `batch_size`, call `transform(batch)` on each chunk (which returns a list), and return all the results concatenated together in order.
Why would an ETL pipeline process a huge dataset in BATCHES rather than loading everything into memory and transforming it all at once?
You can implement batched processing that keeps memory usage bounded regardless of total dataset size, the foundation of any real ETL pipeline.