Feed-Forward Network.

Feed-forward networks are the multilayer perceptron (MLP) components within each transformer block that process each token position independently after the attention mechanism. These networks typically consist of two linear transformations with a non-linear activation function in between, expanding the hidden dimension significantly before projecting back down. Despite being conceptually simple, feed-forward networks contain the majority of parameters in transformer models and are crucial for the model's capacity to learn complex patterns and transformations.

The feed-forward network operates on each token position separately, unlike attention which considers relationships between tokens. It typically expands the hidden dimension by a factor of 4 or more (often 8 in modern models), applies a non-linear activation like ReLU or GELU, then projects back to the original dimension. This expansion allows the network to learn rich, non-linear transformations of the token representations. The position-wise nature means the same feed-forward network is applied to every token position, but each processes its input independently.

Feed-forward networks represent a significant portion of computational cost during both training and inference, often accounting for 60-80% of total parameters in large language models. This parameter density makes them prime targets for optimization techniques like mixture of experts, which can reduce computational requirements by activating only subsets of the feed-forward capacity. A common misconception is that attention mechanisms do most of the 'thinking' in transformers, when in reality the feed-forward networks perform much of the actual knowledge storage and transformation work.