Vocabulary.

Vocabulary represents the complete dictionary of tokens that a language model has learned to understand and produce during training. This fixed set determines what linguistic elements the model can process, from common words to rare terms, punctuation, and special symbols. The vocabulary size directly impacts the model's ability to handle diverse languages, technical terminology, and out-of-vocabulary words, making it a crucial architectural decision that affects both capability and efficiency.

Modern language models use subword tokenization methods like Byte Pair Encoding (BPE) or SentencePiece to balance vocabulary size with coverage. Larger vocabularies can represent more concepts directly but require more memory and computational resources, while smaller vocabularies force longer token sequences for the same text. The vocabulary is typically learned from training data through statistical analysis of token frequency, with rare tokens often merged or represented through subword combinations to maintain manageable vocabulary sizes.

Vocabulary size creates fundamental trade-offs in model design and deployment. Larger vocabularies enable better handling of specialized domains and multilingual content but increase memory requirements and inference costs. Smaller vocabularies reduce resource needs but may struggle with rare words, proper nouns, or code, often requiring multiple tokens to represent single concepts. Understanding these constraints helps developers choose appropriate models for specific use cases and explains why some models excel in certain domains while struggling in others.