- RAG gives LLMs dynamic context from retrieved information to overcome training data limitations and knowledge cutoffs.
- RAG works through indexing documents into vector embeddings and retrieving similar chunks to ground LLM responses.
- HyDE, reranking, multi-hop retrieval, and contextual retrieval address basic RAG's synthesis and reasoning limitations.
- RAG fails through retrieval misses, chunk fragmentation, context overload, and faithfulness issues despite retrieved context.
- RAGAS measures faithfulness, answer relevancy, and context precision/recall for production RAG quality assessment.
- RAG suits dynamic knowledge and citations while fine-tuning handles consistent style and domain reasoning patterns.
Why RAG Exists
LLMs know what was in their training data, but not what happened after their cutoff date, what's in your private documents, or what's currently on the internet. RAGRAGRetrieval-Augmented Generation — a technique that enhances LLM responses by first retrieving relevant documents from an external knowledge base and including them in the prompt.Learn more → solves this by giving the model a dynamic context windowContext WindowThe maximum amount of text (measured in tokens) that a model can read and reason over in a single interaction, including your prompt, any documents, and previous conversation history.Learn more → filled with retrieved information relevant to each specific query, rather than relying solely on the static knowledge encoded in model weights.
RAG was introduced in a 2020 Facebook AI Research paper and has become the default architecture for enterprise AI applications. Document Q&A, customer support bots, internal knowledge bases, and research assistants are all typically built on RAG pipelines. It is now so fundamental that 'RAG or fine-tune?' is a standard question in AI system design.
How a RAG Pipeline Works
A basic RAG pipeline has two phases: indexing and retrieval-augmented generation. Indexing: chunk documents into segments, embed each chunk using an embedding modelEmbedding ModelA specialized neural network that converts text into high-dimensional vectors, enabling semantic search, clustering, and retrieval applications.Learn more →, store the embeddings in a Vector DatabaseVector DatabaseA database optimized for storing and searching embedding vectors, enabling fast semantic similarity search across millions of text chunks or documents.Learn more →. Generation: embed the user's query, search the Vector Database for the most similar chunks, retrieve those chunks, insert them into the LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more →'s promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more → as context, generate a response citing the retrieved information. This GroundingGroundingThe process of connecting an LLM's outputs to verified external information sources to improve accuracy and reduce hallucinations.Learn more → step is what distinguishes RAG from bare Foundation ModelFoundation ModelA large AI model trained on broad data at massive scale that can be adapted to a wide variety of downstream tasks, forming the 'foundation' for specialized applications.Learn more → calls — retrieved facts anchor the response and prevent HallucinationHallucinationWhen an AI model generates text that is factually incorrect, fabricated, or unsupported by its context, stated with confident fluency as if it were true.Learn more → on private or recent information.
The key insight is that similarity in embeddingEmbeddingA numerical vector representation of text, code, or images that captures semantic meaning — similar items have similar vectors, enabling search, clustering, and retrieval.Learn more → space correlates with semantic relevance. Asking 'What is the refund policy?' will retrieve the most semantically similar chunks from your knowledge base — ideally the chunks that actually contain the refund policy — even if they don't contain the exact words from the query.
Beyond Basic RAG: Advanced Techniques
Basic RAG struggles with: questions requiring synthesis across many documents, queries whose answers span multiple chunks, complex multi-hop reasoning, and situations where the relevant information isn't obviously similar to the query. Advanced techniques address these: HyDE (generating a hypothetical answer and embedding that for retrieval), rerankingRerankingA second-stage model that reorders retrieved candidates by relevance to improve the quality and precision of RAG system outputs.Learn more → (using a cross-encoderEncoderThe transformer component that processes input sequences into contextual representations, forming the foundation of understanding-focused models like BERT.Learn more → to rerank retrieved chunks), and multi-hop retrieval (iteratively retrieving based on intermediate reasoning steps).
Query expansion, parent document retrieval, and sentence window retrieval are practical improvements that dramatically help real-world performance. Contextual retrieval — Anthropic's technique of prepending context summaries to each chunk before embedding — significantly improves retrieval quality by making chunks more self-contained.
RAG Failure Modes
RAG can fail in several ways: retrieval failures (the relevant chunk isn't found because the query and document are phrased differently), chunk fragmentation (the answer requires information from multiple chunks that aren't retrieved together), context overload (too many retrieved chunks dilute the model's attention), and faithfulness failures (the model ignores retrieved context and hallucينates anyway).
Evaluation is the hardest part of production RAG. RAGAS is the most popular evaluation framework — it measures faithfulness (is the answer supported by context?), answer relevancy (does the answer address the question?), and context precision/recall (were the right chunks retrieved?). Running regular RAGAS evaluations as you iterate is essential for maintaining quality.
RAG vs Fine-Tuning: How to Choose
RAG is better when: your knowledge base changes frequently, you need source citations, you have large amounts of content, or privacy requires keeping data out of training. Fine-tuningFine-tuningThe process of further training a pre-trained model on a smaller, task-specific dataset to specialize its behavior for a particular use case.Learn more → is better when: you need consistent format or style, you want to teach domain-specific reasoning patterns, or you need to reduce latencyLatencyThe delay between sending a request to an LLM and receiving the first token of the response, often measured as Time to First Token (TTFT).Learn more → by using a smaller model trained on domain data.
The most powerful systems often combine both: fine-tune for style and task-specific reasoning, use RAG for factual grounding and up-to-date knowledge. Many production systems also use both in parallel — fine-tuned models for pattern matching and speed, RAG for accuracy on knowledge-intensive queries.