Grouped-Query Attention.

Grouped-Query Attention (GQA) is a memory optimization technique for transformer models that reduces the computational and storage overhead of the attention mechanism during inference. Unlike standard multi-head attention where each attention head has its own key and value projections, GQA groups multiple query heads to share the same key and value projections. This architectural change significantly reduces the size of the key-value (KV) cache that models must maintain during text generation, making inference faster and more memory-efficient while preserving most of the model's performance.

The technique works by organizing attention heads into groups where each group shares a single set of key and value weight matrices, while maintaining separate query projections for each head within the group. This creates a middle ground between standard multi-head attention (where every head has unique K, V, and Q projections) and multi-query attention (where all heads share the same K and V projections). The grouping strategy allows models to maintain much of the representational capacity of full multi-head attention while dramatically reducing memory requirements, particularly for the KV cache that grows with sequence length during autoregressive generation.

GQA offers an attractive trade-off for production deployments where inference speed and memory efficiency are critical constraints. The technique typically achieves substantial reductions in KV cache size and inference latency with minimal impact on model quality, making it particularly valuable for serving large language models at scale. However, the optimal number of groups and group sizes depend on the specific model architecture and use case, requiring careful tuning. Some practitioners mistakenly assume GQA always improves performance, but the benefits are most pronounced in memory-constrained scenarios or when serving long sequences where the KV cache becomes a bottleneck.