Semantic vs keyword search
Keyword search (BM25, Elasticsearch) finds documents containing the exact words in the query. A search for 'vehicle maintenance' misses documents that talk about 'car servicing' or 'automobile repair'. Semantic searchSemantic SearchSemantic search retrieves information based on meaning and context using vector embeddings rather than exact keyword matches.Learn more → embeds both the query and documents into a shared vector space, finding matches based on meaning regardless of word choice.
The two approaches are complementary. Hybrid search — combining BM25 scores with semantic similarity scores — outperforms either method alone. For most production systems, implement semantic search first, then add keyword search and blend the scores using Reciprocal Rank Fusion (RRF).
Build the indexing pipeline
Step 1: Load documents. `from langchain.document_loaders import DirectoryLoader; docs = DirectoryLoader('./docs', glob='**/*.md').load()`.
Step 2: Split into chunks. Use `RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64)`. Chunk size of 512 tokens balances specificity (small enough to match a specific answer) and context (large enough to be coherent). The 64-tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → overlap prevents losing context at chunk boundaries.
Step 3: Embed all chunks. Use `SentenceTransformer('BAAI/bge-m3').encode([c.page_content for c in chunks])`. Store embeddings alongside the chunk text and metadata in a numpy array or vector databaseVector DatabaseA database optimized for storing and searching embedding vectors, enabling fast semantic similarity search across millions of text chunks or documents.Learn more →.
Build the search function
For small corpora (<100K documents), FAISS is the fastest in-memory vector search: `import faiss; index = faiss.IndexFlatIP(embedding_dim); index.add(embeddings_array); distances, indices = index.search(query_embedding.reshape(1, -1), k=10)`.
Return the top-kTop-KA sampling parameter that restricts token selection to the k most probable next tokens, discarding the long tail of unlikely options.Learn more → chunks sorted by similarity score. Apply a minimum threshold (e.g. 0.75 cosine similarityCosine SimilarityA metric that measures similarity between two vectors by calculating the cosine of the angle between them, widely used in vector search and retrieval systems.Learn more →) to filter irrelevant results. If no results exceed the threshold, fall back to a keyword search or return an 'I don't know' response.
Re-rank for precision
The top-k results from vector search are close in semantic space but not necessarily ranked by relevance to the exact query. Re-ranking with a cross-encoderEncoderThe transformer component that processes input sequences into contextual representations, forming the foundation of understanding-focused models like BERT.Learn more → model significantly improves precision at the top of the results.
Use a cross-encoder: `from sentence_transformers import CrossEncoder; reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L6-v2'); scores = reranker.predict([(query, chunk) for chunk in top_k_chunks]); ranked = sorted(zip(scores, top_k_chunks), reverse=True)`. The re-ranker processes each (query, chunk) pair together and produces a more accurate relevance score.
Use the retrieved results
For a question-answering system, take the top 3–5 re-ranked chunks and inject them into an LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more →: 'Use the following passages to answer the question. If the answer is not in the passages, say so. Passages: [chunks]. Question: [query].'
Always include source citations in the response. Surface the document name and page number alongside each cited fact so users can verify the source. This builds trust and makes errors easier to detect.