Streaming.

Streaming is a response delivery method where large language models output tokens one by one as they generate them, rather than waiting to complete the entire response before sending it to the user. This approach dramatically improves the perceived responsiveness of AI applications by allowing users to see text appearing in real-time, similar to watching someone type. Streaming is particularly valuable for longer responses where waiting for completion would create noticeable delays and poor user experience.

The streaming process works by establishing a persistent connection between the client and server, typically using Server-Sent Events (SSE) or WebSocket protocols. As the model generates each token through its inference process, it immediately transmits that token to the client application, which displays it to the user. This differs from traditional request-response patterns where the entire output is buffered server-side before transmission. The implementation requires careful handling of connection management, error states, and graceful termination when generation completes.

Streaming significantly reduces time-to-first-token perception and keeps users engaged during longer generations, but introduces complexity in application design and error handling. Developers must account for partial responses, connection interruptions, and the inability to modify earlier parts of streamed content. While streaming improves user experience, it doesn't actually reduce the total generation time or computational cost—it simply redistributes when users receive the information. Some applications may prefer non-streaming responses for batch processing or when the complete response is needed before taking action.