Residual Connection.

Residual connections are architectural components in neural networks that create shortcuts by adding a layer's input directly to its output, forming the expression output = f(x) + x where f(x) is the layer's transformation and x is the input. This technique was introduced to solve the vanishing gradient problem that plagued very deep networks, where gradients become exponentially smaller as they propagate backward through many layers during training. Residual connections enable the training of networks with hundreds or even thousands of layers by providing direct pathways for gradients to flow.

The key insight behind residual connections is that they allow layers to learn residual functions rather than complete transformations. Instead of learning a mapping H(x), layers learn the residual F(x) = H(x) - x, making it easier to optimize identity mappings when needed. During backpropagation, gradients can flow directly through these skip connections without being diminished by layer transformations, ensuring that even the earliest layers receive meaningful gradient signals. This architectural choice fundamentally changed how deep networks are designed and trained.

Residual connections have become ubiquitous in modern deep learning architectures, appearing in ResNets, Transformers, and most state-of-the-art models including large language models. While they enable much deeper networks, they also increase memory usage and computational overhead due to the additional connections and operations. A common misconception is that residual connections simply prevent vanishing gradients, but they also help with optimization dynamics and can improve feature reuse across layers, making networks more parameter-efficient and easier to train.