KV Cache.
A memory optimization that stores previously computed attention keys and values so the model doesn't recompute them when generating each new token.
During inference, a transformer model computes attention across all tokens in the context for every new token it generates. Without a KV (key-value) cache, this requires re-processing the entire input sequence repeatedly — an O(n²) operation that becomes prohibitively slow for long contexts.
The KV cache stores the key and value matrices computed during the initial processing of the prompt. When generating the next token, the model only needs to compute attention for the new token against the cached keys and values, reducing redundant computation dramatically. This is why context length affects memory requirements more than compute for generation.
KV cache memory is a significant constraint in serving large models with long contexts. A 1M-token context with a large model can require hundreds of gigabytes of KV cache memory. Research into efficient KV cache management — including quantization, eviction policies, and offloading — is an active area of systems work.