Token.

The basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.

Tokens are the fundamental units of text that language models process. A tokenizer splits input text into a sequence of tokens before feeding it to the model. Common English words are typically single tokens. Longer or less common words may be split into multiple subword tokens. A rough rule of thumb: 1 token ≈ 0.75 English words, or about 4 characters.

Tokenization is not purely word-based. GPT-style models use Byte Pair Encoding (BPE) tokenization, which splits text at the subword level based on frequency patterns in training data. The word 'tokenization' might be a single token; 'antidisestablishmentarianism' would be split into several. Numbers, code, and non-English text often tokenize less efficiently than English prose.

Tokens are also the unit of API pricing. When an LLM API charges $3 per million input tokens, every character in your prompt, documents, and conversation history contributes to that count. Output tokens (the generated response) are typically priced separately and higher. Understanding tokenization helps optimize prompts for cost and context window usage.