Context Windows
18 min
Every LLM has a fixed context window — a maximum number of tokens it can attend to at once. A document longer than that has to be split into chunks. Splitting into non-overlapping windows is the naive approach, but it risks cutting an idea exactly in half at a chunk boundary, so the model never sees that content in one coherent window. The fix is a sliding window with overlap: each new window starts partway through the previous one, so boundary content still shows up whole in at least one chunk.
def chunk_tokens(tokens, window_size, stride):
chunks = []
i = 0
while i < len(tokens):
chunk = tokens[i:i + window_size]
chunks.append(chunk)
if i + window_size >= len(tokens):
break
i += stride
return chunks
stride == window_size (the first example) gives you plain non-overlapping chunking -- overlap only appears once stride is smaller than window_size, which is exactly the knob you'd tune to trade 'more redundant compute' for 'less risk of splitting content awkwardly.'
Write `chunk_tokens(tokens, window_size, stride)`: split `tokens` into overlapping windows of length `window_size`, advancing `stride` tokens between window starts. Include a final, possibly-shorter window if any tokens remain, then stop. Return the list of windows.
Why would you chunk a long document into OVERLAPPING windows (stride < window_size) instead of non-overlapping ones, when feeding it to a model with a limited context window?
You can implement sliding-window chunking with configurable overlap, the standard technique for fitting long documents through a model's fixed context window.