Streaming Responses
16 min
Without streaming, an application has to wait for the model to finish generating the ENTIRE response before showing anything — for a long answer, that can feel sluggish even if total generation time is identical. Streaming sends the response incrementally as small delta chunks, each containing just the next few tokens of text, so the UI can render output as it arrives — the "typing" effect familiar from chat interfaces.
def accumulate_stream(chunks):
text = ""
for chunk in chunks:
content = chunk.get("delta", {}).get("content")
if content:
text += content
return text
A real streaming client (via Server-Sent Events) processes these chunks one at a time AS THEY ARRIVE over the network, updating the UI after each one -- accumulate_stream here just simulates receiving the whole sequence at once and reconstructing the final text, the same reconstruction step a real client does incrementally.
Write `accumulate_stream(chunks)`: each chunk is a dict like `{'delta': {'content': '...'}}`, where `'content'` may be missing or `None` (e.g. the very first chunk just announces the role). Concatenate every present, non-empty `content` value in order and return the full accumulated text.
Why would an application request a STREAMING response instead of waiting for the complete chat completion?
You can reconstruct a complete message from streamed delta chunks, and understand why streaming improves perceived responsiveness without changing total generation time.