Attention Mechanism.
A technique that allows each token in a sequence to 'pay attention' to all other tokens, enabling the model to understand context and relationships across long distances.
The attention mechanism, introduced in the landmark 2017 paper 'Attention Is All You Need', is the core innovation behind the Transformer architecture. It allows a model to weigh the relevance of every token to every other token in a sequence, rather than processing text linearly like older RNNs did.
In practice, attention works by computing three vectors for each token: a Query (what this token is looking for), a Key (what this token represents), and a Value (what information this token provides). The dot product of queries and keys determines attention weights, which are then used to create a weighted sum of values.
Modern LLMs use multi-head attention, where many attention operations run in parallel, each focusing on different aspects of the relationships between tokens. This allows the model to simultaneously capture syntactic structure, semantic meaning, coreference, and factual associations.