How to build semantic search from scratch

Semantic search finds relevant results based on meaning rather than keywords. This guide builds a complete semantic search system from document ingestion to ranked retrieval, using embeddings and a vector store.

Build the indexing pipeline

Step 1: Load documents. `from langchain.document_loaders import DirectoryLoader; docs = DirectoryLoader('./docs', glob='**/*.md').load()`.

Step 2: Split into chunks. Use `RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64)`. Chunk size of 512 tokens balances specificity (small enough to match a specific answer) and context (large enough to be coherent). The 64-tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → overlap prevents losing context at chunk boundaries.

Step 3: Embed all chunks. Use `SentenceTransformer('BAAI/bge-m3').encode([c.page_content for c in chunks])`. Store embeddings alongside the chunk text and metadata in a numpy array or vector databaseVector DatabaseA database optimized for storing and searching embedding vectors, enabling fast semantic similarity search across millions of text chunks or documents.Learn more →.

Build the search function

For small corpora (<100K documents), FAISS is the fastest in-memory vector search: `import faiss; index = faiss.IndexFlatIP(embedding_dim); index.add(embeddings_array); distances, indices = index.search(query_embedding.reshape(1, -1), k=10)`.

Return the top-kTop-KA sampling parameter that restricts token selection to the k most probable next tokens, discarding the long tail of unlikely options.Learn more → chunks sorted by similarity score. Apply a minimum threshold (e.g. 0.75 cosine similarityCosine SimilarityA metric that measures similarity between two vectors by calculating the cosine of the angle between them, widely used in vector search and retrieval systems.Learn more →) to filter irrelevant results. If no results exceed the threshold, fall back to a keyword search or return an 'I don't know' response.

Re-rank for precision

The top-k results from vector search are close in semantic space but not necessarily ranked by relevance to the exact query. Re-ranking with a cross-encoderEncoderThe transformer component that processes input sequences into contextual representations, forming the foundation of understanding-focused models like BERT.Learn more → model significantly improves precision at the top of the results.

Use a cross-encoder: `from sentence_transformers import CrossEncoder; reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L6-v2'); scores = reranker.predict([(query, chunk) for chunk in top_k_chunks]); ranked = sorted(zip(scores, top_k_chunks), reverse=True)`. The re-ranker processes each (query, chunk) pair together and produces a more accurate relevance score.

Use the retrieved results

For a question-answering system, take the top 3–5 re-ranked chunks and inject them into an LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more →: 'Use the following passages to answer the question. If the answer is not in the passages, say so. Passages: [chunks]. Question: [query].'

Always include source citations in the response. Surface the document name and page number alongside each cited fact so users can verify the source. This builds trust and makes errors easier to detect.