Sliding-Window Attention.

Sliding-window attention is an optimization technique that restricts each token in a sequence to only attend to a fixed number of nearby tokens, rather than all previous tokens in the sequence. This approach dramatically reduces the computational complexity of attention from quadratic to linear with respect to sequence length, making it feasible to process much longer contexts. The technique is particularly valuable for applications requiring long-context understanding while maintaining computational efficiency.

The mechanism works by defining a window size (such as 512 or 1024 tokens) and allowing each token to only compute attention weights with tokens within that window. As the model processes the sequence, this window slides forward, ensuring that each position maintains access to recent context while discarding older information. Some implementations use overlapping windows or combine sliding-window attention with other patterns like global attention on special tokens to maintain some long-range dependencies.

While sliding-window attention enables processing of much longer sequences at lower computational cost, it comes with trade-offs in modeling capability. The fixed window size means that tokens cannot directly attend to information beyond the window boundary, potentially limiting the model's ability to capture very long-range dependencies. However, through multiple layers, information can still propagate across the entire sequence. This technique has become essential for making transformer models practical for long-context applications like document processing and extended conversations.