- Google Brain researchers introduced the transformer architecture in June 2017 with 'Attention Is All You Need' paper.
- Modern LLMs use decoder-only (GPT, Llama), encoder-only (BERT), or encoder-decoder (T5) transformer variants.
- Computes Query, Key, and Value vectors for each token to model arbitrary dependencies regardless of distance.
- Uses 32-128 parallel attention heads that learn different relationship types like grammar and semantics.
- Modern LLMs use RoPE to inject position information, enabling configurable context windows and sequence understanding.
- Transformers process all tokens simultaneously unlike sequential RNNs, enabling massive GPU-based scale-up.
Attention Is All You Need
In June 2017, eight Google Brain researchers published a paper titled 'Attention Is All You Need.' It introduced the TransformerTransformerThe neural network architecture that underpins virtually all modern LLMs, introduced in 2017, built around self-attention mechanisms that process entire sequences in parallel.Learn more → — an architecture that replaced recurrent connections entirely with self-attention. The claim seemed bold: could a model with no sequential processing truly understand language? The results were decisive. The transformer outperformed every existing architecture on translation benchmarks and trained in a fraction of the time.
The impact of this paper is difficult to overstate. GPTGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →, BERT, T5, Claude, Gemini, Llama — every major AI system of the past seven years is built on the foundations laid in those pages. Understanding the transformer is understanding the engine of the AI revolution.
Encoder vs Decoder Transformers
The original transformer had two components: an encoderEncoderThe transformer component that processes input sequences into contextual representations, forming the foundation of understanding-focused models like BERT.Learn more → (which reads input text and creates contextual representations) and a decoderDecoderA transformer component that generates text tokens sequentially using causal attention, where each token can only attend to previous tokens in the sequence.Learn more → (which generates output tokens, attending to both the encoder's output and previously generated tokens). This architecture is natural for translation: the encoder reads French, the decoder generates English.
Modern LLMs predominantly use decoder-only architectures (GPT, Llama, Claude, Gemini). Decoder-only models are trained to predict the next tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → in a sequence, which makes them natural generative models. BERT and its variants use encoder-only architectures, which excel at understanding and classification tasks. T5 uses the full encoder-decoder setup, performing well on tasks that can be framed as sequence-to-sequence.
Self-Attention: The Core Mechanism
Self-attention works by computing three vectors for each token: Query (Q), Key (K), and Value (V). These are linear projections of the token's embeddingEmbeddingA numerical vector representation of text, code, or images that captures semantic meaning — similar items have similar vectors, enabling search, clustering, and retrieval.Learn more →. Attention scores are computed as the dot product of each Query with every Key, scaled by the square root of the key dimension, then passed through softmaxSoftmaxA mathematical function that converts a vector of raw logits into a probability distribution, with temperature controlling the sharpness of the output.Learn more → to create probability weights. The output for each token is the weighted sum of all Values.
Intuitively: the Query asks 'what am I looking for?', the Key says 'what do I represent?', and the Value says 'what information do I provide?' The dot product of Q and K determines relevance; high relevance high weight on that K's corresponding V. This is why the attention mechanismAttention MechanismA technique that allows each token in a sequence to 'pay attention' to all other tokens, enabling the model to understand context and relationships across long distances.Learn more → can model arbitrary dependencies between tokens regardless of their distance.
Multi-Head Attention
Modern transformers use multi-head attentionMulti-Head AttentionA mechanism that runs multiple attention operations in parallel, each learning different aspects of relationships between tokens.Learn more →: the Q, K, V projections are split into multiple 'heads' that run attention in parallel, each in a lower-dimensional subspace. Different heads learn to attend to different types of relationships — one head might track grammatical subject-verb agreement, another coreference, another semantic similarity.
The outputs of all heads are concatenated and projected back to the model dimension. Modern frontier models use 32–128 attention heads. Grouped-Query AttentionGrouped-Query AttentionAn attention optimization technique where multiple attention heads share key and value projections in groups, reducing memory usage and speeding up inference while maintaining model quality.Learn more → (GQA), used by Llama 3 and others, reduces the number of K/V heads while keeping Q heads at full count, cutting the KV cacheKV CacheA memory optimization that stores previously computed attention keys and values so the model doesn't recompute them when generating each new token.Learn more → memory requirements without significantly impacting quality.
Positional Encoding
Attention is order-agnostic — without positional information, the model can't distinguish 'dog bites man' from 'man bites dog.' Positional encodings inject position information into the embeddings. The original transformer used sinusoidal encodings; modern LLMs use RoPE (Rotary Position EmbeddingRotary Position EmbeddingRotary Position Embedding (RoPE) encodes positional information by rotating query and key vectors, enabling attention to depend on relative distances between tokens.Learn more →), which encodes position through rotations in embedding space and generalizes well to sequences longer than those seen during training.
Positional encodingPositional EncodingA technique that adds position information to tokens in transformer models, enabling them to understand word order despite attention being inherently position-agnostic.Learn more → is what allows transformers to have configurable context windows. Models can be extended to handle longer sequences by adjusting their positional encoding scheme — though performance typically degrades beyond the training length. Research into better positional encodings is an active area, with YaRN and ALiBi showing promise for long-context extrapolation.
Why Transformers Displaced Everything Else
Before transformers, LSTMs and RNNs were the dominant sequence models. They processed text token by token, maintaining a hidden state that compressed previous context. This sequential processing couldn't be parallelized, making training slow. Long-range dependencies were difficult to maintain as information had to travel through many steps.
Transformers process all tokens simultaneously, directly attending between any two positions in O(n²) operations. This is both their strength (direct long-range modeling) and weakness (quadratic memory in sequence length). The parallelism maps perfectly to GPU matrix operations, enabling the massive scale-up that produced today's frontier models. Alternative architectures like Mamba and RWKV attempt to recover the efficiency of sequential models while matching transformer quality — an active research competition.