Two ways to get a model's answer: wait for the whole response at once, or receive it token by token as it is generated. They differ mostly in one metric your users actually feel — time-to-first-token.
What each mode does
- Non-streaming — you send the request and get the complete response in one payload when the model finishes.
- Streaming — the server sends tokens as they are generated, and your client renders them immediately.
Under the hood both produce the same total tokens. The difference is when you see them.
How streaming works under the hood
Streaming rides on server-sent events (SSE). The server keeps the HTTP connection open and writes each token as a small data: event as it is produced. Your client consumes the stream incrementally rather than waiting for one big JSON body. The protocol is simple — the complexity is all in the client side, where you assemble partial text into a usable state.
Why streaming wins on interactivity
In non-streaming mode, a user stares at a blank screen until the entire answer is complete. A 500-token reply might take several seconds to arrive in full, and every one of those seconds feels like dead time.
Streaming shows the first token the instant it is ready — often a few hundred milliseconds — and then draws the rest. Perceived latency drops dramatically even though the total time is roughly the same.
For any interactive chat surface, stream. The user experience difference is the difference between "fast" and "broken."
Time-to-first-token is the metric that matters
The number to watch is not total request time — it is time-to-first-token (TTFT): how long until the first piece of the answer appears. A user tolerates a long answer taking a while as long as something is happening. Measure TTFT in your logs, and you will see exactly where a model or provider is hurting the experience: a fast total time with a slow TTFT still feels broken, because the user was staring at nothing for most of it.
When non-streaming is the better call
Streaming is not always worth the extra client complexity:
- Batch and background jobs — no human is waiting, so a single response is simpler.
- One-shot API calls — extraction, classification, or summarization where the caller just needs the result.
- Strict parsers — when you need the full JSON in one piece rather than assembling it from deltas.
- Server-to-server flows — where reconstructing a stream is unnecessary work.
In these cases, non-streaming means less code and fewer failure modes for identical output.
Handling tool calls and JSON in a stream
One real gotcha: structured output arrives awkwardly in streaming mode. Tool-call arguments and JSON are often streamed as fragment strings that you must buffer and parse at the end of the stream. If your app depends on calling a function or parsing JSON, you will either buffer the relevant chunks or fall back to non-streaming for those specific calls.
The practical trade-offs
Streaming is not free:
- Your client must handle partial responses, connection drops, and mid-stream errors.
- Functions and tool calls often arrive at the end of the stream, so you may need to buffer.
- Some providers price or quota streaming differently — verify before assuming they are identical.
None of these are hard, but they are real work you should budget for only where the interactivity pays off.
Pick by surface
A simple rule covers most apps: stream on anything a person reads in real time, and use non-streaming for anything a machine consumes. See how to toggle stream and handle server-sent events in the API reference, and check per-model latency in the model catalog to tune your time-to-first-token.