How prompt caching works
Processing a promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more → from scratch requires computing a key-value (KV) representation for every tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more →. Prompt cachingPrompt CachingA technique that stores and reuses computed results from repeated prompt prefixes across API requests, reducing both response latency and computational costs.Learn more → stores this KV cacheKV CacheA memory optimization that stores previously computed attention keys and values so the model doesn't recompute them when generating each new token.Learn more → on the server after the first request. When a subsequent request starts with the same tokens, the provider reuses the cached computation — skipping the expensive prefillPrefillThe initial inference phase where an LLM processes the entire input prompt in parallel to populate the KV cache before beginning sequential token generation.Learn more → step.
Caching is most effective when you have a large, repeated prefix: a lengthy system promptSystem PromptA special instruction given to a language model before the user conversation begins, establishing the model's persona, capabilities, constraints, and context.Learn more →, a long document being queried multiple times, or a large codebase being analysed. Short or highly variable prompts see little benefit.
Anthropic cache_control
Anthropic's API exposes caching via the `cache_control` field on message content blocks. Mark the part of your prompt you want cached: add `{"cache_control": {"type": "ephemeral"}}` to the content block. Anthropic caches the KV state up to and including that block for 5 minutes.
Example — cache a long system document: `messages=[{"role": "user", "content": [{"type": "text", "text": LONG_DOCUMENT, "cache_control": {"type": "ephemeral"}}, {"type": "text", "text": "Summarise the key points."}]}]`. The cached prefix write costs 25% more but reads cost 90% less. Break-even is at 2+ reads of the same prefix.
OpenAI automatic caching
OpenAI automatically caches prompts longer than 1024 tokens for up to 1 hour. No code changes are required — just ensure your repeating prefix is at least 1024 tokens. Cache hits reduce your input token cost by 50%. Check `response.usage.prompt_tokens_details.cached_tokens` to verify caching is working.
Structure your prompts to put the stable content first: system prompt static documents dynamic user input. The cache applies to the longest matching prefix, so the more stable content at the start, the higher the cache hit rate.
Structuring prompts for maximum cache hits
The golden rule: put stable content at the top, variable content at the bottom. If you have a 5000-token system prompt followed by a 100-token user query, the 5000-token prefix will be cached across all requests regardless of the query.
For multi-turn conversations, cache the system prompt plus the entire conversation history up to the previous turn. Only the latest user message is dynamic. With a 20-turn conversation, caching the history saves approximately 90% of the input token cost from turn 2 onward.