KV Cache: How LLMs Remember Context Efficiently

The key-value cache is the mechanism that lets LLMs process long conversations without recomputing everything from scratch on every token.

Key takeaways
  • Stores key and value attention matrices to avoid recomputing attention over all previous tokens during generation.
  • Reduces generation complexity from O(n²) to O(n) work per token by reusing cached vectors.
  • Can require tens of gigabytes per request for large models, often exceeding model weight memory usage.
  • Quantized caches, sliding window attention, and grouped-query attention help reduce memory consumption.
  • Reuses cached KV representations across requests sharing common prefixes like system prompts or documents.
  • Cached prefix tokens are billed at up to 90% discount from regular input token prices.

The Problem the KV Cache Solves

During autoregressiveAutoregressiveA text generation approach where models produce sequences one token at a time, with each new token conditioned on all previously generated tokens.Learn more → text generation, every new tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → must attend to all previous tokens. Without caching, generating the nth token would require recomputing attention over all n-1 previous tokens — O(n²) work for an n-token response. For long documents or multi-turn conversations, this would make generation prohibitively slow and expensive.

The key-value (KV) cache solves this by storing the key and value attention matrices for every token as they are computed. When generating a new token, the model reuses cached K and V vectors from all previous tokens instead of recomputing them. Only the new token's K/V vectors need to be computed, reducing generation to O(n) work per token.

Memory Tradeoffs

The KV cacheKV CacheA memory optimization that stores previously computed attention keys and values so the model doesn't recompute them when generating each new token.Learn more → dramatically speeds up generation but consumes substantial GPU memory. For a 70B parameter model with a 128K context windowContext WindowThe maximum amount of text (measured in tokens) that a model can read and reason over in a single interaction, including your prompt, any documents, and previous conversation history.Learn more →, the KV cache can require tens of gigabytes of memory per concurrent request — often more than the model weights themselves. This memory pressure is a primary constraint on how many simultaneous requests an inferenceInferenceThe process of running a trained AI model to generate outputs — what happens when you send a prompt and receive a response.Learn more → server can handle.

Techniques like quantized KV caches (storing K/V in 4-bit or 8-bit instead of 16-bit), sliding window attention (only caching the most recent N tokens), and grouped-query attentionGrouped-Query AttentionAn attention optimization technique where multiple attention heads share key and value projections in groups, reducing memory usage and speeding up inference while maintaining model quality.Learn more → (GQA, which reduces the number of K/V heads) all help compress KV cache memory. These tradeoffs directly affect how much context length is practical at a given cost.

Prefix Caching

Prefix caching (also called prompt cachingPrompt CachingA technique that stores and reuses computed results from repeated prompt prefixes across API requests, reducing both response latency and computational costs.Learn more →) extends the KV cache idea across requests: if many queries share the same prefix (a system promptSystem PromptA special instruction given to a language model before the user conversation begins, establishing the model's persona, capabilities, constraints, and context.Learn more →, document, or code file), the provider computes and caches the KV representations for that prefix once, reusing them across all subsequent requests. This dramatically reduces cost and latencyLatencyThe delay between sending a request to an LLM and receiving the first token of the response, often measured as Time to First Token (TTFT).Learn more → for high-prefix applications.

Anthropic, OpenAI, and Google all offer promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more → caching as a feature. Tokens in the cached prefix are billed at a significant discount (up to 90% off input token prices). For applications with long, stable system prompts or shared context, enabling prompt caching is one of the highest-ROI optimizations available.

Read next
6 min read

LLM Inference: How Models Generate Text

Inference is what happens when an LLM produces a response. Understanding it helps you optimize for speed, cost, and quality.

7 min read

Context Windows Explained: The Working Memory of LLMs

What context windows are, why they matter for building AI applications, how they've grown from 4K to 10M tokens, and how to manage them effectively.

9 min read

The Attention Mechanism: How LLMs Understand Context

A clear explanation of self-attention — the mathematical operation at the heart of every transformer that allows language models to understand relationships between words.

7 min read

Tokens and Tokenization: The Building Blocks of LLMs

Everything you need to know about tokens — how LLMs split text into pieces, why tokenization matters for cost and performance, and how different languages tokenize.