- Text is tokenized, embedded into vectors, processed through transformer layers, and sampled token-by-token at 50-100 tokens per second.
- Text becomes integer IDs mapped to high-dimensional vectors (4,096-8,192 dimensions) that encode initial meaning before context.
- Each layer applies multi-head self-attention for context-aware representations, then feed-forward networks with residual connections.
- Information is distributed across billions of floating-point weights in embedding matrices, attention projections, and feed-forward networks.
- Models produce one token at a time left-to-right, unable to revise earlier tokens, with errors potentially propagating through responses.
The LLM Processing Pipeline
When you send a message to an LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more →, a series of transformations occurs before text appears in response. First, your text is broken into tokens — subword units that the model processes. These tokens are converted to numerical vectors (embeddings) that enter a stack of transformerTransformerThe neural network architecture that underpins virtually all modern LLMs, introduced in 2017, built around self-attention mechanisms that process entire sequences in parallel.Learn more → layers. Each layer refines the representations through attention and feed-forward operations. Finally, the output layer produces a probability distribution over the vocabularyVocabularyThe fixed set of tokens (words, subwords, characters) that a language model can recognize and generate, typically ranging from thousands to millions of unique elements.Learn more →, and a tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → is sampled. This process repeats for every token in the response.
The full pipeline: Text Tokenizer Token embeddings Transformer layers (×N) Output logitsLogitsRaw, unnormalized numerical scores that language models output for each possible next token before applying softmax to convert them into probabilities.Learn more → SamplingSamplingThe method used to select each next token from a probability distribution during text generation, controlling the randomness and creativity of model outputs.Learn more → Token Detokenizer Text. Each step happens in milliseconds on modern hardware, which is why you see responses streamingStreamingStreaming returns a model's tokens incrementally as they're generated rather than waiting for the complete response to finish.Learn more → at 50–100 tokens per secondTokens per SecondTokens per second measures how quickly an LLM generates text during inference, serving as the primary throughput metric for comparing model speed.Learn more →.
Tokens and Embeddings
Tokens are the atoms of LLM processing. A tokenizer maps text to integer IDs: 'Hello world' might become [15496, 995]. The model never sees raw text — only these integers. An embeddingEmbeddingA numerical vector representation of text, code, or images that captures semantic meaning — similar items have similar vectors, enabling search, clustering, and retrieval.Learn more → layer maps each integer to a high-dimensional vector (typically 4,096–8,192 dimensions for large models). These vectors encode the initial meaning of each token before context is applied.
The embedding dimension is a fundamental architectural choice. Higher-dimensional embeddings can encode richer information but require more memory and compute. The embeddings are learned during training — by the end of training, semantically similar tokens have similar embedding vectors, even if their surface forms are completely different.
What Happens Inside a Transformer Layer
A transformer layer applies two operations in sequence. First, multi-head self-attention: each token's representation is updated based on weighted information from all other tokens in the context. The attention weights determine how much each token 'attends to' each other token — creating context-aware representations. Second, a feed-forward networkFeed-Forward NetworkThe position-wise multilayer perceptron within each transformer block that processes tokens individually after attention, containing most of the model's parameters.Learn more →: a two-layer MLP applied independently to each position, adding non-linearity and enabling the model to transform representations further.
Residual connections wrap both operations, meaning each layer's output is added to its input rather than replacing it. This is crucial for training deep networks — gradients flow back through the residual path, enabling effective learning across dozens or hundreds of layers. Layer normalizationLayer NormalizationA normalization technique that standardizes activations across features within each layer to stabilize training and improve convergence speed.Learn more → stabilizes training by normalizing activations before each operation.
Where 'Knowledge' Lives
An LLM's 'knowledge' is not stored in a database or lookup table. It is encoded in billions of floating-point numbers — the model's weights — distributed across embedding matrices, attention projections, and feed-forward networks. This distributed, implicit storage is what makes LLMs both powerful (they compress patterns from vast data) and unreliable (they can confabulate with fluent confidence).
The feed-forward layers in particular appear to act as 'memory' banks. Research has shown that factual associations are often stored in specific FFN neurons and can be surgically modified — a technique called model editing. The model doesn't 'know' that Paris is the capital of France the way a database does; it has learned that these tokens tend to co-occur in ways consistent with that fact.
How Text Is Generated
Generation is autoregressiveAutoregressiveA text generation approach where models produce sequences one token at a time, with each new token conditioned on all previously generated tokens.Learn more →: the model produces one token at a time, with each new token appended to the context before generating the next. This creates a left-to-right dependency chain. The output logits (unnormalized scores for each vocabulary token) are transformed by temperatureTemperatureA parameter controlling the randomness of model outputs — lower values produce more focused, deterministic responses; higher values produce more creative, varied text.Learn more → and sampling parametersParametersThe numerical weights inside a neural network that are learned during training — the 'knowledge' of the model, measured in billions for modern LLMs.Learn more → (top-pTop-PAlso called nucleus sampling — a method that selects from the smallest set of tokens whose combined probability exceeds a threshold p, adapting dynamically to the model's confidence at each step.Learn more →, top-kTop-KA sampling parameter that restricts token selection to the k most probable next tokens, discarding the long tail of unlikely options.Learn more →) into a probability distribution, and a token is drawn from that distribution.
This process has implications for LLM behavior: the model cannot revise earlier tokens, so errors early in a response can propagate and compound. It also means the model 'commits' to a reasoning path as it generates, which is why techniques like Chain of ThoughtChain of ThoughtA prompting technique that encourages an LLM to reason step by step before giving a final answer, dramatically improving performance on complex tasks.Learn more → prompting and multi-step reasoning (which allocate explicit tokens to thinking) improve accuracy on hard tasks. The Foundation ModelFoundation ModelA large AI model trained on broad data at massive scale that can be adapted to a wide variety of downstream tasks, forming the 'foundation' for specialized applications.Learn more → underlying a chat assistant is also what powers Agents — autonomous systems that call tools and reason across multiple steps.