How to handle rate limits and retries in LLM applications

Rate limit errors are the most common production issue in LLM applications. This guide covers exponential backoff, token budget management, request queuing, and provider-specific rate limit strategies.

Understanding rate limits

LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → APIs rate limitRate LimitRestrictions on the number of API requests or tokens that can be processed within a specific time period to manage server capacity and costs.Learn more → on two dimensions: requests per minute (RPM) and tokens per minute (TPM). RPM limits cap the number of API calls; TPM limits cap the total input + output tokens. Exceeding either returns a 429 Too Many Requests error. TPM limits are more commonly hit than RPM limits in production.

Rate limits vary enormously by provider tier. OpenAI's tier 1 (newly funded accounts) has very low limits; tier 5 (high-spend accounts) has limits orders of magnitude higher. Check your current limits at platform.openai.com/account/limits. Plan your architecture around your current tier, not your expected final tier.

Implement exponential backoff

Never retry immediately after a 429. Use exponential backoff with jitter: wait 1 second after the first failure, 2 after the second, 4 after the third, up to a maximum of 60 seconds. Add random jitter (±20%) to prevent all clients from retrying simultaneously after an outage.

Python implementation: `import time, random; def call_with_backoff(fn, max_retries=5): for i in range(max_retries): try: return fn(); except RateLimitError: if i == max_retries - 1: raise; wait = (2**i) + random.uniform(0, 1); time.sleep(wait)`. The `tenacity` library provides a production-quality decorator for this.

Manage your token budget

Track tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → usage per request and maintain a rolling counter against your TPM limit. The OpenAI response includes `usage.total_tokens` — accumulate this and throttle requests when approaching the limit.

Set `max_tokens` on every request. Without it, a model might generate 4096 tokens when 200 would suffice, burning through your token budget. For most user-facing applications, 500–1000 max output tokens is appropriate. For coding tasks, 2000–4000 is reasonable.

Use a request queue for burst handling

In applications with many concurrent users, implement a request queue with rate limiting. Libraries like `asyncio-throttle` (Python) or `p-limit` (Node.js) limit concurrent API calls. Process the queue at your safe request rate, rejecting or queuing excess requests with appropriate user feedback.

For background processing jobs (batch summarisation, nightly analysis), use the OpenAI Batch API which processes requests asynchronously at half the price with a 24-hour completion window. Submit a JSONL file of requests and poll for completion.

Multi-provider fallback

The most robust production setup routes requests across multiple providers. If OpenAI returns a 429, retry with Anthropic; if Anthropic is slow, fall back to Gemini. OpenRouter handles this automatically — set multiple fallback models in your request: `{"models": ["openai/gptGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →-4o", "anthropic/claude-sonnet-4-5"], "route": "fallback"}`.

Cache responses for identical prompts. Many production applications repeat the same or similar prompts — caching saves both cost and latencyLatencyThe delay between sending a request to an LLM and receiving the first token of the response, often measured as Time to First Token (TTFT).Learn more →. Use a semantic cache (hash the embeddingEmbeddingA numerical vector representation of text, code, or images that captures semantic meaning — similar items have similar vectors, enabling search, clustering, and retrieval.Learn more → of the promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more →) to catch near-duplicate requests. Redis or PostgreSQL with pgvector work well for this.