- Fundamental text unit that LLMs process as integer IDs mapped from subword pieces.
- GPT models use Byte Pair Encoding with 50,000-100,000 token vocabularies from character merging.
- API pricing charges per million tokens for input and output text processing.
- English tokenizes most efficiently while Chinese and Arabic use 3-4× more tokens.
- Model limits measured in tokens, with 128K tokens holding roughly 96,000 English words.
- Poor tokenization of numbers and non-English text degrades model arithmetic and language abilities.
What Is a Token?
A tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → is the fundamental unit of text that a language model processes. Before an LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → reads your text, a tokenizer splits it into tokens — subword units that are mapped to integer IDs. The model never sees raw characters; it sees a sequence of these integer IDs, which are then converted to numerical vectors for processing.
In English, tokens roughly correspond to words or word parts. Common words like 'the', 'is', 'and' are typically single tokens. Longer or less common words get split: 'tokenizationTokenizationThe process of breaking raw text into discrete units (tokens) that language models can process, typically using subword algorithms like BPE.Learn more →' might become ['token', 'ization']. Numbers, punctuation, and whitespace each tokenize differently. A useful rule of thumb: 1 token ≈ 4 characters ≈ 0.75 English words.
Byte Pair Encoding: How GPT Tokenizes
OpenAI's models use Byte Pair Encoding (BPE). BPE starts by splitting text into individual characters, then iteratively merges the most frequent adjacent pairs into single tokens. Trained on a large corpus, BPE produces a vocabularyVocabularyThe fixed set of tokens (words, subwords, characters) that a language model can recognize and generate, typically ranging from thousands to millions of unique elements.Learn more → of typically 50,000–100,000 tokens that efficiently represents the language's common patterns.
The GPTGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →-4 tokenizer (cl100k_base) has 100,257 tokens. You can explore any text's tokenization using OpenAI's Tokenizer playground. Seeing how your text is tokenized is illuminating — 'ChatGPT is great!' tokenizes to ['Chat', 'G', 'PT', ' is', ' great', '!'], not word-by-word as you might expect.
Why Tokenization Matters for You
Token count directly determines API cost. LLM APIs charge per million tokens for both input (your promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more →) and output (the response). A 1,000-word document is roughly 1,300 tokens. Sending 10,000 such documents to GPT-4o at $2.50/M input tokens costs about $32.50. Understanding token efficiency helps control costs at scale.
Tokenization also affects performance. Some tasks encode poorly in tokens: large numbers (1,234,567 becomes ['1', ',', '234', ',', '567'] — 5 tokens instead of 1 meaning), code in uncommon languages, and non-English text. Models struggle more with arithmetic partly because numbers tokenize inefficiently, breaking the model's number sense.
How Tokenization Differs Across Languages
English is the most efficiently tokenized language in most models because their training data is English-heavy. Chinese, Japanese, and Arabic text tokenizes far less efficiently — a Chinese sentence might use 3–4× as many tokens as the equivalent English sentence. This means non-English API usage is proportionally more expensive and context windows accommodate fewer words.
Newer models and tokenizers are improving multilingual efficiency. Llama 3's tokenizer (128K vocabulary) is significantly better for non-English languages than Llama 2's. Qwen's tokenizer is specifically optimized for Chinese. Multilingual tokenization efficiency is an important consideration for global applications.
Tokens and Context Windows
A model's context windowContext WindowThe maximum amount of text (measured in tokens) that a model can read and reason over in a single interaction, including your prompt, any documents, and previous conversation history.Learn more → is measured in tokens, not words or characters. A 128,000-token context window holds roughly 96,000 English words — about 300 pages. This sounds generous, but context fills up quickly when you include system prompts, conversation history, retrieved documents, and tool outputs.
Effective context management is a key skill in building LLM applications. Strategies include semantic chunkingChunkingThe process of splitting documents into smaller passages for embedding and retrieval in RAG systems, where chunk size and overlap affect retrieval quality.Learn more → (splitting documents at natural boundaries), summarization (compressing earlier conversation history), and selective retrieval (only including the most relevant document sections). Every token in context costs money and can affect model quality.