Byte-Pair Encoding.

Byte-Pair Encoding (BPE) is a data compression technique adapted for natural language processing that creates subword vocabularies by iteratively identifying and merging the most frequent pairs of adjacent characters or bytes. Originally developed for text compression, BPE became foundational for modern language models because it elegantly solves the vocabulary size problem—enabling models to handle rare words and out-of-vocabulary terms without requiring massive dictionaries. This approach allows models to break down unknown words into familiar subword components, making tokenization both efficient and robust.

The BPE algorithm starts with a base vocabulary of individual characters or bytes, then repeatedly finds the most frequent pair of adjacent symbols in the training corpus and merges them into a single token. This process continues for a predetermined number of iterations, gradually building larger subword units from smaller ones. Unlike word-level tokenization that struggles with rare terms, or character-level tokenization that creates very long sequences, BPE finds a middle ground by learning the optimal subword boundaries from data. The resulting vocabulary contains both common whole words and meaningful subword fragments that can be recombined to represent any text.

BPE's main advantage is its ability to handle the open vocabulary problem while keeping sequence lengths manageable, which directly impacts model efficiency and performance. However, BPE can sometimes create suboptimal splits, particularly across morphological boundaries or in languages with complex word formation rules. Modern variants like SentencePiece have refined the approach, but the core BPE principle remains central to most contemporary language models. Understanding BPE is crucial for anyone working with language models, as tokenization choices significantly affect model behavior, training efficiency, and the handling of different languages and domains.