Continuous Batching.

Continuous batching is a serving optimization technique that addresses the inefficiencies of traditional static batching in large language model inference. Unlike static batching where all requests in a batch must complete before new ones can be processed, continuous batching allows the system to dynamically add new requests to a running batch as soon as slots become available when other requests finish. This approach is crucial for maximizing GPU utilization and throughput in production LLM deployments, where requests naturally have varying completion times due to different output lengths and complexity.

The technique works by maintaining a dynamic pool of active requests on the GPU, continuously monitoring for completed sequences and immediately filling vacant slots with queued requests. This requires sophisticated memory management and scheduling algorithms that can handle variable-length sequences and different attention patterns within the same batch. Key technical challenges include managing the key-value cache efficiently across requests of different lengths, handling memory fragmentation, and ensuring that the attention computation remains correct when batch composition changes dynamically during execution.

Continuous batching can significantly improve serving throughput compared to static batching, often achieving 2-10x higher request processing rates depending on the workload characteristics. However, it introduces implementation complexity and requires careful tuning of batch size limits, memory allocation strategies, and request scheduling policies. A common misconception is that continuous batching always reduces latency for individual requests, but while it improves overall system throughput, individual request latency may sometimes increase due to resource sharing with a larger number of concurrent requests.