Prompt Caching.

Prompt caching is an optimization technique used by LLM providers to store the computational results of processing common prompt prefixes, allowing subsequent requests with the same prefix to skip redundant computation. This approach significantly reduces both response latency and costs for applications that repeatedly use similar prompt structures, such as system prompts, few-shot examples, or lengthy context documents. The technique leverages the fact that transformer models process tokens sequentially, so identical prefixes produce identical intermediate states that can be cached and reused.

The caching mechanism works by storing the key-value pairs and attention states generated during the processing of a prompt prefix, typically identified by a hash of the input tokens. When a new request arrives with a matching prefix, the cached computation is retrieved and the model only needs to process the new tokens that follow the cached portion. This creates substantial efficiency gains for applications with consistent prompt templates, though it requires the prefix to match exactly at the token level. Different providers implement varying cache retention policies, from minutes to hours, balancing memory usage with performance benefits.

Prompt caching delivers the most value for applications with stable, reusable prompt components like system instructions, documentation contexts, or standardized few-shot examples, potentially reducing costs by 50-90% for the cached portions. However, it requires careful prompt design to maximize cache hit rates, as even small changes to the prefix invalidate the cache. Common misconceptions include expecting caching to work with dynamic prefixes or assuming all providers offer the same caching capabilities, when in reality cache duration, pricing models, and implementation details vary significantly across different LLM services.