- Model selection, prompt efficiency, caching, output control, and batching can achieve 50-80% total cost reduction when applied systematically.
- Using appropriate models for request complexity instead of expensive models for all requests captures 60% of savings with 20% of routing effort.
- Anthropic and OpenAI charge cached tokens at 10-25% of standard rates, reducing input costs by 70-90% for applications with long system prompts.
- Output tokens cost 3-5× more than input tokens, making length constraints and structured formats high-leverage cost reduction strategies.
- Tools like LLM-cache achieve 30-50% hit rates for similar queries in customer support applications by reusing previous responses.
- Batch API endpoints offer 50% cost reduction in exchange for 24-hour response time for non-real-time workloads.
The Five Cost Levers
LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → costs have five main levers: model selection (60–90% reduction potential), promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more → efficiency (20–40% reduction), caching (30–70% reduction for repeated content), output control (15–30% reduction), and batching (10–30% reduction). Most cost-optimization projects achieve 50–80% total reduction by working through all five systematically.
Audit your current spending first. Break down cost by: model, use case, prompt length distribution, output length distribution, and error rate. Most applications have a 20% of requests that drive 80% of cost — optimizing those high-cost patterns first gives the fastest returns.
Model Routing: Right Model for Each Request
Using GPTGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →-5 for every request when GPT-4o Mini handles 70% of them adequately is the most common and expensive cost mistake. Implement routing: classify request complexity, route simple requests to cheap models, escalate complex ones. A routing classifier itself can be a tiny, cheap model — or even simple heuristics (short prompt + common category = small model).
Start with manual routing (define categories and assign models), then measure quality per category, then automate. The first 20% of routing effort typically captures 60% of savings. Perfect routing is expensive and usually not worth it — rough routing based on obvious features gets most of the benefit.
Prompt Caching
Anthropic and OpenAI both offer prompt cachingPrompt CachingA technique that stores and reuses computed results from repeated prompt prefixes across API requests, reducing both response latency and computational costs.Learn more → — if you send the same prompt prefix on multiple requests, cached tokens are charged at 10–25% of standard input rates. For applications with long system prompts (knowledge base excerpts, detailed instructions, few-shotFew-ShotA prompting technique where you include a small number of input-output examples in the prompt to demonstrate the desired task format and behavior to the model.Learn more → examples), caching can reduce input costs by 70–90%.
Structure your prompts to maximize caching: put the stable, cacheable content (system promptSystem PromptA special instruction given to a language model before the user conversation begins, establishing the model's persona, capabilities, constraints, and context.Learn more →, long instructions, reference documents) at the beginning, and variable content (user messages, dynamic context) at the end. Cache hits are guaranteed only when the prefix is byte-identical, so avoid random elements or timestamps in cacheable sections.
Output Length Control
Output tokens typically cost 3–5× more than input tokens per unit. Controlling output length is high-leverage. Use `max_tokens` to cap output when appropriate. Specify output format explicitly: 'Respond in 2–3 sentences maximum' or 'Return a JSON object with these fields only.' Models without length constraints often generate verbose responses that add cost without adding value.
Ask for structured outputStructured OutputConstraining a language model to produce output that conforms to a predefined format or schema, such as valid JSON, rather than free-form text.Learn more → (JSON, numbered lists, tables) rather than flowing prose when you're processing output programmatically. Structured formats use fewer tokens than equivalent prose and are easier to parse. JSON modeJSON ModeA model setting that constrains LLM output to syntactically valid JSON format, ensuring parseable responses without full schema validation.Learn more → or function callingFunction CallingA capability that allows language models to return structured calls to developer-defined functions or tools, enabling AI systems to interact with external APIs, databases, and services.Learn more → eliminates wrapper text that you'd strip anyway.
Semantic Caching and Batch Processing
Semantic caching reuses previous responses for semantically similar queries — 'What is your return policy?' and 'Can I return a product?' may be answered identically. Tools like LLM-cache and GPTCache implement semantic caching using embeddingEmbeddingA numerical vector representation of text, code, or images that captures semantic meaning — similar items have similar vectors, enabling search, clustering, and retrieval.Learn more → similarity. Effective hit rates of 30–50% are common for customer support and FAQ applications.
For batch processing tasks (document analysis, data enrichment, content generation), batch API endpoints from Anthropic and OpenAI offer 50% cost reduction in exchange for 24-hour response time. If you don't need real-time responses for a workload, batch processing is a simple, significant saving.