How to build a RAG pipeline from scratch

Retrieval-Augmented Generation (RAG) extends LLMs with your own documents, giving accurate, source-grounded answers without fine-tuning. This guide builds a working RAG system from document ingestion to response generation.

What RAG solves

LLMs have a knowledge cutoffKnowledge CutoffThe date after which a language model has no training knowledge of world events, absent external tools or retrieval systems.Learn more → and cannot know about your private documents. RAGRAGRetrieval-Augmented Generation — a technique that enhances LLM responses by first retrieving relevant documents from an external knowledge base and including them in the prompt.Learn more → solves this by retrieving relevant text chunks from a document store at query time and injecting them into the promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more → as context. The model then answers based on the retrieved context rather than its training data.

RAG is ideal for: Q&A over internal documentation, customer support bots with a knowledge base, code search over a large codebase, and any application where accuracy and sourcing are critical.

The four components of a RAG pipeline

1. Ingestion: Load documents, split them into chunks, convert each chunk into a vector embeddingEmbeddingA numerical vector representation of text, code, or images that captures semantic meaning — similar items have similar vectors, enabling search, clustering, and retrieval.Learn more →, and store the embeddings in a vector databaseVector DatabaseA database optimized for storing and searching embedding vectors, enabling fast semantic similarity search across millions of text chunks or documents.Learn more →.

2. Retrieval: When a query arrives, embed the query with the same model, search the vector database for the N most similar chunks, and return them.

3. Augmentation: Inject the retrieved chunks into the prompt as context, along with the original query.

4. Generation: Pass the augmented prompt to the LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → and return the response.

Build the ingestion pipeline

Install dependencies: `pip install langchain openai chromadb tiktoken pypdf`. Load PDFs: `from langchain.document_loaders import PyPDFLoader; docs = PyPDFLoader('document.pdf').load()`. Split into chunks: `from langchain.text_splitter import RecursiveCharacterTextSplitter; splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200); chunks = splitter.split_documents(docs)`.

Create embeddings and store in ChromaDB: `from langchain.vectorstores import Chroma; from langchain.embeddings import OpenAIEmbeddings; vectorstore = Chroma.from_documents(chunks, OpenAIEmbeddings(), persist_directory='./chroma_db')`. Call `vectorstore.persist()` to save to disk.

Build the retrieval and generation pipeline

Load the persisted vectorstore: `vectorstore = Chroma(persist_directory='./chroma_db', embedding_function=OpenAIEmbeddings())`. Create a retriever: `retriever = vectorstore.as_retriever(search_kwargs={'k': 5})`.

Build the RAG chain with LangChain: `from langchain.chains import RetrievalQA; from langchain.chat_models import ChatOpenAI; qa_chain = RetrievalQA.from_chain_type(llm=ChatOpenAI(model='gptGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →-4o-mini'), chain_type='stuff', retriever=retriever, return_source_documents=True)`.

Query: `result = qa_chain({'query': 'What is the refund policy?'}); print(result['result']); print(result['source_documents'])`. The response includes citations to the source chunks.

Improve retrieval quality

Hybrid search: combine dense vector search with BM25 keyword search. Dense search finds semantically similar content; BM25 finds exact keyword matches. Combining both with a reranker reduces retrieval failures.

Chunk overlap is critical for maintaining context at boundaries — use `chunk_overlap=200` to ensure sentences split at chunk boundaries are included in both chunks. Too little overlap leads to fragmented context; too much wastes tokens.

Add metadata filtering: tag each chunk with its source document, section, and page number. Let users filter by source and include metadata in the context so the model can cite specific pages.