Why streaming matters for UX
A 500-tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → response at 40 tok/s takes 12.5 seconds without streamingStreamingStreaming returns a model's tokens incrementally as they're generated rather than waiting for the complete response to finish.Learn more →. With streaming, the user sees the first words in under a second. Research on user perception shows that responses that start appearing within 1 second feel 'instant' regardless of total generation time.
Streaming is not just a UX feature — it also allows users to stop generation early if the model is going in the wrong direction, reducing wasted API spend.
How streaming works technically
LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → APIs stream responses using Server-Sent Events (SSE), a simple HTTP protocol where the server keeps the connection open and sends `data: {...}\n\n` events as tokens are generated. The client reads the stream line by line and processes each event.
Each event in the OpenAI format is a JSON object with a `choices[0].delta.content` field containing the new token(s). The final event is `data: [DONE]\n\n`, signalling the end of the stream.
Express backend with streaming
On the server, set SSE headers: `res.setHeader('Content-Type', 'text/event-stream'); res.setHeader('Cache-Control', 'no-cache'); res.setHeader('Connection', 'keep-alive'); res.flushHeaders();`.
Stream from OpenAI: `const stream = await openai.chat.completions.stream({...}); for await (const chunk of stream) { const token = chunk.choices[0]?.delta?.content; if (token) { res.write('data: ' + JSON.stringify({token}) + '\n\n'); } } res.write('data: [DONE]\n\n'); res.end();`
React frontend consuming SSE
Use the `EventSource` API or the `fetch` API with a readable stream. The `fetch` approach is more flexible: `const response = await fetch('/api/stream', {method: 'POST', body: JSON.stringify({promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more →})}); const reader = response.body.getReader(); const decoderDecoderA transformer component that generates text tokens sequentially using causal attention, where each token can only attend to previous tokens in the sequence.Learn more → = new TextDecoder();`
Read chunks in a loop: `while (true) { const {done, value} = await reader.read(); if (done) break; const text = decoder.decode(value); const lines = text.split('\n'); for (const line of lines) { if (line.startsWith('data: ') && line !== 'data: [DONE]') { const data = JSON.parse(line.slice(6)); setMessage(prev => prev + data.token); } } }`
Handle errors and cancellation
Implement abort on the client: create an `AbortController`, pass `signal: controller.signal` to `fetch`, and call `controller.abort()` when the user cancels. On the server, listen for the `close` event on the response object and abort the OpenAI stream.
Always wrap stream reading in a try-finally block. If the client disconnects mid-stream, you will get an error — catch it gracefully rather than letting it propagate. Log the error for monitoring but do not crash the process.