How to monitor and reduce LLM API costs in production

LLM API costs can grow unexpectedly as usage scales. This guide covers cost attribution, per-request tracking, anomaly detection, and practical techniques that reduce costs by 50–80% without sacrificing quality.

Track costs per request and user

Every LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → API response includes tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → usage. Log these alongside the model, feature, and user ID on every call: `{model, input_tokens, output_tokens, cost_usd, feature, user_id, timestamp}`. This gives you cost attribution at any level of granularity — by user, by feature, by model.

Calculate cost from token counts: maintain a pricing table keyed by model. Example: `PRICING = {'gptGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →-4o': {'input': 2.50, 'output': 10.00}}` (per million tokens). `cost = (input_tokens * PRICING[model]['input'] + output_tokens * PRICING[model]['output']) / 1e6`.

Set up cost alerts

Configure spending alerts at the provider dashboard — this is your last-resort safety net. But also implement application-level alerts: if daily spend exceeds 1.5× the 7-day average, page the on-call engineer. Sudden spikes usually indicate a bug (infinite loop, runaway agentAgentAn LLM-powered system that can take actions, use tools, and pursue multi-step goals autonomously without human input at each step.Learn more →) or unexpectedly viral traffic.

Implement per-user spending limits for consumer applications. Track cumulative daily cost per user. When a user exceeds their limit (e.g. $1/day on a free tier), return an error and promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more → upgrade. This prevents any single user from consuming disproportionate resources.

Reduce input costs with caching

Repeated prompts (same user asking follow-up questions, same system promptSystem PromptA special instruction given to a language model before the user conversation begins, establishing the model's persona, capabilities, constraints, and context.Learn more → used across requests) are the biggest cost optimisation opportunity. Implement response caching for identical prompts: hash the full prompt and store the response in Redis with a 24-hour TTL.

Enable provider-side prompt cachingPrompt CachingA technique that stores and reuses computed results from repeated prompt prefixes across API requests, reducing both response latency and computational costs.Learn more → (Anthropic cache_control, OpenAI automatic caching) to reduce the cost of repeated prefixes. For a system with a 2000-token system prompt sent with every request, prompt caching can reduce input token costs by 80%.

Right-size your model selection

Not every request needs GPT-4o. Classify requests by complexity and route to the appropriate model: use a fast, cheap model (GPT-4o-mini, Claude Haiku, Gemini 2.0 Flash) for simple tasks like extraction, classification, and reformatting; use a capable model only for tasks that genuinely require it (complex reasoning, code generation, nuanced writing).

Implement a model cascade: try the cheap model first, check if the output meets your quality bar (via a fast assertion), and only escalate to the expensive model if needed. For many applications, 70–80% of requests are satisfied by the cheap model, reducing average cost per request by 60–70%.

Optimise prompts for token efficiency

Audit your system prompts for redundancy. Teams often accumulate instructions over time — 'be helpful, be concise, never make up information, always cite sources, use markdown, use bullet points...' Many can be removed or condensed. Every 100 tokens removed from the system prompt saves cost on every request.

Compress retrieved context in RAGRAGRetrieval-Augmented Generation — a technique that enhances LLM responses by first retrieving relevant documents from an external knowledge base and including them in the prompt.Learn more → pipelines. Instead of including full document chunks, summarise them first with a cheap model and inject the summaries. A 500-token chunk summarised to 100 tokens is 80% cheaper as context while retaining the key information.