Serving Patterns
22 min
Running a model on hardware like a GPU is usually far more efficient per-item when processing many inputs at once (a batch) than one at a time — the hardware's parallelism goes largely unused on a single input. A model server exploits this with dynamic batching: instead of running inference the instant each request arrives, it waits briefly to accumulate a batch, bounded by both a maximum size (don't grow the batch forever) and a maximum wait time (don't make the first request in the batch wait too long for later ones to show up).
def batch_requests(requests, max_batch_size, max_wait):
batches = []
current_batch = []
batch_start = None
for req_id, arrival in requests:
if not current_batch:
current_batch = [req_id]
batch_start = arrival
elif len(current_batch) < max_batch_size and (arrival - batch_start) <= max_wait:
current_batch.append(req_id)
else:
batches.append(current_batch)
current_batch = [req_id]
batch_start = arrival
if current_batch:
batches.append(current_batch)
return batches
Both stopping conditions matter independently: max_batch_size caps GPU memory usage per batch, max_wait caps how long the FIRST request in a batch has to sit around waiting for company -- a real server tunes both based on its latency SLA and hardware.
Write `batch_requests(requests, max_batch_size, max_wait)`: `requests` is a list of `(request_id, arrival_time)` sorted by arrival time. Group requests into batches: start a new batch's clock at the first request's arrival time; keep adding requests to the CURRENT batch as long as it's under `max_batch_size` AND the request arrives within `max_wait` of the batch's start time; otherwise start a new batch. Return a list of batches (each a list of request IDs).
Why does a model server use DYNAMIC batching (grouping several incoming requests before running inference) instead of running each request through the model immediately, one at a time?
You can implement dynamic request batching bounded by both size and wait time, the mechanism model servers use to trade a little latency for much higher throughput.