Streaming (LLM)

Streaming is the technique of sending model output token-by-token to the user as it generates — making the response feel fast even when total time is unchanged.

A 2-second wait followed by an instant block of text feels worse than the same 2-second wait with text appearing word-by-word from the start. Streaming makes AI products feel responsive at no cost to actual latency. All major providers ship streaming on chat completions.

Example

ChatGPT streams. The full answer takes 8 seconds, but the first words appear in 0.4s — and the user perceives 'fast'.

How OpenLabor uses it

OpenLabor streams every conversational surface so the user sees thinking happen in real time.

Does streaming work with structured outputs?

Partially. Most providers stream JSON, but parsers must handle incremental valid-but-incomplete state.

Related: latency, llm.

AI Labor Glossary