Tokenization.

Tokenization is the fundamental preprocessing step that converts human-readable text into discrete units called tokens that language models can understand and process. Rather than working with individual characters or whole words, modern LLMs use subword tokenization methods like Byte Pair Encoding (BPE) or SentencePiece that break text into meaningful chunks. This approach balances vocabulary size with semantic representation, allowing models to handle both common words and rare terms efficiently while maintaining reasonable computational requirements.

The tokenization process typically involves splitting text into subword units based on frequency patterns learned from training data, creating a fixed vocabulary of tokens that the model maps to numerical representations. Different tokenization schemes handle various languages and scripts with varying efficiency - English text might tokenize into roughly 3-4 characters per token on average, while languages with different writing systems may require more tokens per character. The choice of tokenizer affects how well models understand different languages, with some tokenizers being more efficient for certain scripts than others.

Tokenization directly impacts both the cost and performance of LLM usage, as models are typically priced per token processed and have fixed context window limits measured in tokens. Inefficient tokenization can lead to higher costs and reduced effective context length, particularly for non-English languages or specialized domains with unique vocabularies. Understanding tokenization helps developers optimize their applications by choosing appropriate models for their target languages, estimating costs accurately, and designing prompts that use tokens efficiently.