- Maximum amount of text an LLM can consider at once, including prompts, history, and documents.
- Context limits expanded from GPT-3's 4K tokens to Meta's Llama 4 Scout reaching 10 million tokens by 2025.
- LLMs recall information near the start and end of context more reliably than content in the center.
- Long contexts require hundreds of gigabytes of GPU memory and quadratically more expensive attention computation.
- Place important information at start or end, summarize histories, and use RAG for selective document retrieval.
- Smaller contexts work with RAG systems while 100K+ contexts enable whole-document analysis at higher cost.
What Is a Context Window?
The context windowContext WindowThe maximum amount of text (measured in tokens) that a model can read and reason over in a single interaction, including your prompt, any documents, and previous conversation history.Learn more → is the maximum amount of text an LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → can consider at once. Everything the model reads — your system promptSystem PromptA special instruction given to a language model before the user conversation begins, establishing the model's persona, capabilities, constraints, and context.Learn more →, conversation history, any documents you paste in, and all previous responses — must fit within this limit. Tokens beyond the limit are simply not visible to the model.
Think of it as working memory. A human reading a very long document will remember the beginning well, follow the argument, and have the ending fresh in mind. The middle may blur. LLMs have an analogous phenomenon called 'lost in the middle' — they recall information near the start and end of the context more reliably than content buried in the center.
The Rapid Growth of Context Windows
Context windows have expanded dramatically. GPTGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →-3 had a 4,096-tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → limit (about 3,000 words). GPT-4 expanded to 32K, then 128K. Claude 2 introduced 100K. By 2025, Claude and GPT models offer 200K tokens; Google's Gemini models offer 1 million tokens; and Meta's Llama 4 Scout reaches an unprecedented 10 million tokens.
Each leap opened new use cases: 8K enabled basic document Q&A; 128K enabled full book analysis; 1M enabled whole codebase understanding; 10M enables processing entire software repositories, years of correspondence, or complete scientific literature in a single promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more →.
The Technical Cost of Long Contexts
Long contextLong ContextThe ability of AI models to process extremely large inputs spanning tens of thousands to millions of tokens, enabling comprehensive analysis of entire documents, codebases, or datasets in a single context.Learn more → windows are expensive in memory and compute. The KV cacheKV CacheA memory optimization that stores previously computed attention keys and values so the model doesn't recompute them when generating each new token.Learn more → — which stores attention key-value pairs for all context tokens — grows linearly with context length. A 1M token context with a large model requires hundreds of gigabytes of GPU memory. This is why long-context queries cost more and why models need specialized infrastructure to serve them efficiently.
Attention computation is also O(n²) in sequence length — quadratically more expensive as context grows. FlashAttentionFlashAttentionAn IO-aware attention algorithm that tiles computation in fast SRAM memory to reduce data movement, enabling efficient training and inference on long sequences.Learn more → and chunked attention implementations dramatically improve this in practice, but serving 1M-token contexts at scale remains a significant engineering challenge. This is why Gemini's 1M context pricing is higher than standard rates.
Practical Context Management
For most applications, effective context management requires being deliberate about what you include. Best practices: put the most important information at the start or end of the context (where retrieval is most reliable), summarize rather than include entire conversation histories for long interactions, use RAGRAGRetrieval-Augmented Generation — a technique that enhances LLM responses by first retrieving relevant documents from an external knowledge base and including them in the prompt.Learn more → to retrieve only relevant document sections rather than pasting everything, and monitor token usage to control costs.
Context window size enables different product designs. Smaller effective contexts work well with retrieval-based architectures (RAG). Very large contexts (100K+) enable whole-document analysis without chunkingChunkingThe process of splitting documents into smaller passages for embedding and retrieval in RAG systems, where chunk size and overlap affect retrieval quality.Learn more →, which eliminates the retrieval problem at the cost of higher per-query cost. The right trade-off depends on your document sizes, query patterns, and budget.