PagedAttention.

PagedAttention is a memory management innovation developed by the vLLM team that applies operating system paging principles to the key-value (KV) cache used in transformer attention mechanisms. Traditional attention implementations allocate contiguous memory blocks for the entire sequence, leading to significant memory fragmentation and waste when sequences vary in length. PagedAttention addresses this by breaking the KV cache into fixed-size pages that can be allocated non-contiguously, dramatically improving memory utilization and enabling higher throughput in production LLM serving.

The system works by dividing attention keys and values into small, fixed-size blocks (typically 16-64 tokens per page) that can be stored anywhere in memory, similar to how operating systems manage virtual memory. When processing sequences, PagedAttention uses a page table to map logical sequence positions to physical memory locations, allowing for dynamic allocation and deallocation of pages as needed. This approach eliminates the need to pre-allocate large contiguous memory blocks and enables features like copy-on-write for shared prefixes between requests, further reducing memory overhead.

PagedAttention enables significantly higher concurrency in LLM serving by reducing memory waste from 60-80% to near zero in many cases, allowing servers to handle more simultaneous requests with the same hardware. The technique is particularly beneficial for variable-length sequences and batch processing scenarios common in production deployments. While PagedAttention adds some computational overhead for memory indirection, this cost is typically negligible compared to the substantial gains in memory efficiency and throughput, making it a critical optimization for cost-effective LLM deployment.