Batch vs. Real-Time Serving
18 min
Batching (from the previous module) isn't free — it's a genuine tradeoff. A bigger batch amortizes the FIXED per-batch overhead (loading the model onto the accelerator, kernel launch cost) over more requests, pushing throughput up. But every request in that batch has to wait for the ENTIRE batch to finish before getting its result, so latency goes up too. A real-time, latency-sensitive service (a live chat autocomplete) wants small batches; a batch-oriented, throughput-sensitive job (nightly scoring of a million rows) wants huge ones.
def throughput_and_latency(batch_size, per_item_compute_ms, fixed_overhead_ms):
total_time_ms = fixed_overhead_ms + batch_size * per_item_compute_ms
throughput = batch_size / (total_time_ms / 1000)
return (round(throughput, 2), total_time_ms)
Notice throughput's gains shrink as batch size grows (66.67 -> 94.12 -> 98.46, diminishing returns) while latency keeps climbing roughly linearly -- past a certain batch size you're mostly just adding latency for very little extra throughput, which is why real systems cap batch size rather than maximizing it.
Write `throughput_and_latency(batch_size, per_item_compute_ms, fixed_overhead_ms)`: total time for a batch is `fixed_overhead_ms + batch_size * per_item_compute_ms`. Return a tuple `(throughput, total_time_ms)` where `throughput` is requests processed per SECOND (`batch_size / (total_time_ms / 1000)`, rounded to 2 decimal places).
Given `serving-patterns`'s batching mechanism, what's the fundamental tradeoff as you increase batch size?
You can quantify the throughput/latency tradeoff of batch size, and know when a workload calls for real-time (small batch) vs. batch-oriented (large batch) serving.