- Embeddings are fixed-length vectors that encode semantic meaning, where similar content has similar vectors.
- Models use contrastive learning with web-scale text pairs to pull similar texts closer and push dissimilar ones apart.
- Key factors include MTEB benchmark scores, vector dimensions, token limits, and cost per million tokens.
- Embeddings are stored in specialized databases like Pinecone or Weaviate for fast approximate nearest-neighbor search.
- Embeddings enable the retrieval layer in RAG systems through query embedding and semantic matching.
What Embeddings Represent
An embeddingEmbeddingA numerical vector representation of text, code, or images that captures semantic meaning — similar items have similar vectors, enabling search, clustering, and retrieval.Learn more → is a fixed-length vector of floating-point numbers that encodes the semantic meaning of a piece of text (or image, or audio). The key property: similar things have similar vectors. 'The quick fox' and 'The fast fox' have very similar embeddings; 'quantum mechanics' and 'pizza recipe' are far apart in embedding space.
This numerical representation of meaning enables mathematics on semantics. You can measure similarity (cosine similarityCosine SimilarityA metric that measures similarity between two vectors by calculating the cosine of the angle between them, widely used in vector search and retrieval systems.Learn more → between vectors), perform analogical reasoning (king - man + woman ≈ queen), and most importantly, search: given a query, find the most semantically similar items in a database — far beyond what keyword matching can achieve.
How Embedding Models Are Trained
Embedding models are trained using contrastive learning: pairs of similar texts are pulled closer together in embedding space; pairs of dissimilar texts are pushed apart. Training data consists of naturally occurring text pairs — questions and their answers, document headings and bodies, translated sentence pairs — at web scale.
Modern embedding models like text-embedding-3-large (OpenAI) and Cohere Embed 3 are bi-encoderEncoderThe transformer component that processes input sequences into contextual representations, forming the foundation of understanding-focused models like BERT.Learn more → models: the query and document are encoded independently, enabling efficient indexing. Cross-encoders process query and document together (more accurate but can't pre-compute document embeddings) and are used for rerankingRerankingA second-stage model that reorders retrieved candidates by relevance to improve the quality and precision of RAG system outputs.Learn more → rather than first-stage retrieval.
Choosing an Embedding Model
Key metrics: MTEB (Massive Text Embedding BenchmarkBenchmarkA standardized test or set of tasks used to evaluate and compare the capabilities of different AI models on a common scale.Learn more →) score (the standard benchmark), vector dimension (higher = more expressive but more storage/compute), max tokensMax TokensThe maximum number of tokens an LLM can generate in a single response, separate from the context window that determines input capacity.Learn more → (how much text can be embedded at once), and cost per million tokens. OpenAI's text-embedding-3-small is an excellent default for English: strong MTEB scores, 1536 dimensions, very low cost.
For multilingual applications, Cohere's multilingual-embed-3 and Voyage AI's multilingual models lead MTEB's multilingual categories. For code retrieval specifically, Voyage Code 2 is the current leader. Domain-specific fine-tuned embeddings (using Sentence Transformers) can significantly outperform general-purpose models for specialized corpora.
Embedding Models and Vector Databases
Embeddings are stored in vector databases — purpose-built infrastructure for fast approximate nearest-neighbor (ANN) search. Pinecone, Weaviate, Qdrant, and Chroma are popular options. For existing PostgreSQL users, the pgvector extension adds vector search without a separate database. Recall at K (fraction of true nearest neighbors returned) is the key quality metric.
The query flow: embed query ANN search in vector databaseVector DatabaseA database optimized for storing and searching embedding vectors, enabling fast semantic similarity search across millions of text chunks or documents.Learn more → (milliseconds, even at billions of vectors) retrieve matching chunks include in LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → context. The embedding modelEmbedding ModelA specialized neural network that converts text into high-dimensional vectors, enabling semantic search, clustering, and retrieval applications.Learn more → determines semantic searchSemantic SearchSemantic search retrieves information based on meaning and context using vector embeddings rather than exact keyword matches.Learn more → quality; the vector database determines search speed and scale. Together they form the retrieval layer of any RAGRAGRetrieval-Augmented Generation — a technique that enhances LLM responses by first retrieving relevant documents from an external knowledge base and including them in the prompt.Learn more → application.