Cross-Attention.

Cross-attention is a fundamental component of transformer architectures that allows one sequence to attend to information from a different sequence. Unlike self-attention where queries, keys, and values all come from the same input, cross-attention creates connections between separate data streams. This mechanism is essential for tasks requiring information integration across different sources, such as machine translation where the decoder must attend to encoder representations, or multimodal applications where text must relate to visual features.

In cross-attention, the query vectors are derived from one sequence (typically the target or decoder sequence) while the key and value vectors come from another sequence (usually the source or encoder sequence). The attention weights are computed by measuring similarity between queries and keys, then used to create weighted combinations of values. This asymmetric structure allows the model to selectively focus on relevant parts of the source sequence when generating each element of the target sequence, creating a learned alignment between different types of information.

Cross-attention enables powerful capabilities like conditional generation, where outputs depend on specific inputs, and multimodal understanding, where models process text, images, and other data types together. However, it introduces computational complexity proportional to the product of sequence lengths and can create bottlenecks in long-context scenarios. A common misconception is that cross-attention always requires different modalities—it's equally important in single-modality tasks like translation where source and target languages represent different sequences within the same domain.