Neural Networks: A Complete Encyclopedia

What Is a Neural Network

A neural network is a computational system loosely inspired by the structure of biological brains. It consists of many simple processing units, called neurons or nodes, connected in layers. Each connection has an associated weight, a number that determines how strongly one neuron influences another. The network learns by adjusting these weights based on the data it processes.

The biological metaphor is only loose. Real neurons are electrochemical devices of extraordinary complexity. Artificial neurons are far simpler mathematical functions. But the high-level organizational principle, many simple units connected in layers, with learning driven by adjusting connection strengths, has proven to be one of the most powerful computational ideas ever discovered.

Neural networks are used because they can learn to perform tasks that are too complex to program explicitly. A computer programmer cannot write explicit rules for recognizing a cat in an image, because the visual features that define 'cat' are too varied and context-dependent to enumerate. But a neural network, trained on millions of images labeled 'cat' and 'not cat,' can learn to recognize cats reliably. The network discovers its own rules from the data.

This ability to learn from data rather than explicit rules is the defining characteristic of neural networks and the broader field of machine learning. It is why neural networks have become the dominant approach in computer vision, natural language processing, speech recognition, and many other domains. Problems that resisted explicit programming for decades yield to the learning-from-data approach.

The field of neural networks has a history that spans more than 75 years, including periods of excitement, disillusionment, and revival. Today, neural networks are the core technology behind some of the most capable and consequential AI systems ever built. Understanding how they work is increasingly essential, not just for researchers and engineers, but for anyone seeking to understand the technology shaping our world.

The Perceptron: The First Building Block

The perceptron was invented by Frank Rosenblatt at the Cornell Aeronautical Laboratory in 1957. It was a hardware implementation of a simple learning algorithm, initially built as a physical machine before being simulated on an IBM 704 computer. Rosenblatt announced it to considerable media fanfare. The New York Times reported that the Navy expected it would be able to walk, talk, see, write, reproduce itself and be conscious of its existence.

The perceptron is a single artificial neuron that takes multiple numerical inputs, multiplies each by a weight, sums the weighted inputs, and produces an output of either 0 or 1 based on whether the sum exceeds a threshold. The learning algorithm adjusts the weights based on whether the perceptron's output was correct or incorrect. If the perceptron misclassified an input, the weights were updated to make the correct classification more likely next time.

The perceptron could learn to classify linearly separable data, meaning datasets where a straight line (or hyperplane in higher dimensions) could separate the two classes. For such problems, the perceptron learning theorem guaranteed that the algorithm would converge to a correct solution in a finite number of steps. This was a remarkable result. A machine that could provably learn to classify data correctly, without being explicitly programmed with the classification rules, was something genuinely new.

The limitations became apparent in 1969, when Marvin Minsky and Seymour Papert published a book called Perceptrons. They proved mathematically that single-layer perceptrons could not learn the XOR function, a simple logical operation, because XOR is not linearly separable. More broadly, they argued that multilayer networks were unlikely to be more powerful. This critique was devastating. Funding for neural network research evaporated almost overnight.

In retrospect, Minsky and Papert's analysis was technically correct but strategically misleading. They had identified a real limitation of single-layer networks. But their pessimism about multilayer networks was premature. The algorithm for training multilayer networks, backpropagation, had not yet been widely understood or appreciated. When it was, it rendered the Minsky-Papert critique obsolete.

Multilayer Networks and Hidden Layers

A multilayer network, also called a multilayer perceptron (MLP) or feedforward neural network, extends the single perceptron into a stack of layers. The first layer, the input layer, receives the raw data. The last layer, the output layer, produces the network's prediction. In between are one or more hidden layers. Each neuron in a hidden layer receives inputs from all neurons in the preceding layer and sends its output to all neurons in the following layer.

The power of multiple layers comes from the ability to learn hierarchical representations. The first hidden layer might learn to detect simple features: edges in an image, common letter combinations in text. The second hidden layer might learn to combine these simple features into more complex ones: shapes from edges, words from letter combinations. Deeper layers learn increasingly abstract representations. This hierarchy mirrors how we understand the brain to process information.

The representation learned by a neural network is distributed. No single neuron corresponds to a specific concept. Instead, each concept is represented by the joint activation pattern of many neurons. This distributed representation is more robust than localist representations where each concept lives in a single neuron. Damage to a few neurons does not destroy entire concepts. It just slightly degrades the representation.

The number of hidden layers defines the depth of a network. Networks with many hidden layers are called deep neural networks, giving rise to the term deep learning. Empirically, deeper networks tend to learn richer, more abstract representations and perform better on complex tasks, provided they can be trained effectively. The challenge of training deep networks was the central problem that deep learning research solved between 2006 and 2015.

The width of a network, the number of neurons in each layer, also matters. Wider networks have more representational capacity. There is a long-standing theorem in mathematics, the universal approximation theorem, which states that a sufficiently wide single-layer network can approximate any continuous function to arbitrary precision. But in practice, deeper networks learn better representations than wider shallow ones for most tasks. Depth is more efficient than width.

Backpropagation: How Networks Learn

Backpropagation is the algorithm that made it possible to train multilayer neural networks. It computes how much each weight in the network contributed to the network's error, allowing all weights to be updated simultaneously to reduce the error. Without backpropagation, training networks with more than one hidden layer was computationally intractable.

The algorithm was described by multiple researchers at different times. Paul Werbos described the key idea in his 1974 PhD thesis. David Rumelhart, Geoffrey Hinton, and Ronald Williams published the landmark paper making it widely known to the AI community in 1986 in the journal Nature. Yann LeCun applied it to convolutional networks for handwritten digit recognition in 1989. It is one of the most important algorithms in the history of computing.

The intuition behind backpropagation is the chain rule of calculus. The network's error depends on the output, which depends on the last hidden layer, which depends on the layer before that, and so on back to the input. Using the chain rule, we can compute how much each weight at each layer contributed to the final error. This gradient information tells us which direction to adjust each weight to reduce the error.

Gradient descent is the optimization algorithm used with backpropagation. Having computed the gradient of the error with respect to each weight, we take a small step in the direction that reduces the error. Repeat this process many times, on many examples, and the weights gradually converge to values that minimize the error on the training data. The learning rate, a small number that controls the size of each step, is a critical hyperparameter that must be tuned carefully.

Modern variants of gradient descent include stochastic gradient descent (SGD), which updates weights based on a single training example or a small batch rather than the entire dataset, and adaptive methods like Adam, which adjust the learning rate for each weight based on the history of gradients. These improvements have made training large networks much more reliable and efficient. But the core idea, compute gradients with backpropagation, take steps with gradient descent, has remained unchanged for 40 years.

Convolutional Neural Networks (CNNs)

Convolutional neural networks are a specialized architecture designed for processing data with a grid-like structure, most notably images. They were pioneered by Yann LeCun in the late 1980s and early 1990s, initially for handwritten digit recognition. They became the dominant approach for computer vision after the AlexNet breakthrough in 2012.

The key operation in a CNN is the convolution. A convolutional layer applies a small filter, typically 3x3 or 5x5 pixels, to every position in the input image. The filter has learned weights that detect a specific feature, such as a horizontal edge, a color gradient, or a specific texture. By sliding the filter across the entire image, the layer produces a feature map showing where that feature appears in the image.

The power of convolution comes from two properties: weight sharing and local connectivity. Weight sharing means the same filter is applied everywhere in the image. This dramatically reduces the number of parameters compared to a fully connected layer where every input pixel has a separate weight for every output neuron. Local connectivity means each output value depends on only a small local patch of the input, which is appropriate because spatial proximity is meaningful in images.

A typical CNN has alternating convolutional layers and pooling layers. Pooling layers, such as max pooling, reduce the spatial resolution of the feature maps by summarizing local regions. This makes the representation more spatially invariant: a feature detected in slightly different positions produces the same activation after pooling. It also reduces the number of computations in subsequent layers.

Deep CNNs learn hierarchical visual representations. Early layers detect edges and textures. Middle layers detect shapes and object parts. Late layers detect entire objects and scenes. This hierarchy emerges automatically from training on labeled image data. No one programs the network to look for edges first. It discovers this strategy on its own because edges are the most informative low-level features for distinguishing visual categories.

Recurrent Networks and LSTMs

Recurrent neural networks (RNNs) were designed to handle sequential data: text, speech, time series, and any other data where order matters. Unlike feedforward networks, which process each input independently, RNNs maintain a hidden state that persists across time steps. Each time the network processes an element of the sequence, it updates its hidden state based on the current input and the previous hidden state.

This gives RNNs a form of memory. The hidden state encodes information about all previous elements in the sequence. In principle, an RNN processing a long sentence can retain information from the beginning of the sentence that is relevant to understanding words at the end. In practice, early RNNs struggled severely with long-range dependencies due to the vanishing gradient problem.

The vanishing gradient problem arises when gradients are propagated backwards through many time steps during training. At each step, the gradient is multiplied by a weight matrix. If these weights are slightly less than one in magnitude, the gradient shrinks exponentially as it travels backward. By the time it reaches the beginning of a long sequence, it is so small that the weights at the beginning receive almost no update signal. The network cannot learn to use information from many steps ago.

Long Short-Term Memory (LSTM) networks, invented by Sepp Hochreiter and Juergen Schmidhuber in 1997, solved the vanishing gradient problem through a different architecture. An LSTM has a cell state, a kind of conveyor belt that runs through the entire sequence, and gates that control what information is written to, read from, and erased from the cell state. These gates allow the network to preserve relevant information over very long sequences without the gradient vanishing.

LSTMs were the dominant architecture for sequence modeling from the late 1990s through 2017. They powered major improvements in speech recognition, machine translation, and language modeling. Google Translate switched to an LSTM-based architecture in 2016, dramatically improving translation quality. But they had a fundamental limitation: they processed sequences one step at a time, which prevented parallelization during training. This made them slow to train on modern GPUs. The transformer architecture, which addressed this limitation, ultimately replaced them.

Attention Mechanisms

The attention mechanism was introduced to address a specific limitation of sequence-to-sequence models: the bottleneck created by encoding an entire input sequence into a single fixed-size vector. In machine translation, for example, this vector had to capture all the information in the source sentence. For long sentences, this was too much information to squeeze into a single representation.

The key insight of attention is that when generating each word of the output, the model should be able to look back at different parts of the input, weighting them by their relevance to the word being generated. When translating a long sentence, different parts of the source sentence are relevant to different parts of the target sentence. Attention allows the model to dynamically focus on the relevant parts at each step.

Bahdanau attention, introduced by Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio in 2015, was the first widely adopted attention mechanism for machine translation. It computed a separate attention score between the current decoder state and each encoder state, then used these scores to create a weighted sum of encoder states that was fed to the decoder. This allowed the decoder to attend to any part of the input at any output step.

Attention dramatically improved machine translation quality, particularly for long sentences. But it was still used within an RNN framework. The encoder and decoder were still recurrent. The computation was still sequential. The step from attention as an auxiliary mechanism within RNNs to attention as the sole mechanism, as in the transformer, was the conceptual leap that enabled the modern AI revolution.

Self-attention is a variant of attention where the queries, keys, and values all come from the same sequence. Rather than attending from a decoder to an encoder, a self-attention layer allows each position in a sequence to attend to all other positions. This enables modeling of long-range dependencies without the sequential bottleneck of recurrence. Multi-head attention applies multiple attention operations in parallel, with different learned projections, allowing the model to attend to different aspects of the relationships simultaneously.

Transformers: The Architecture That Changed Everything

The transformer architecture, introduced in the 2017 paper 'Attention Is All You Need' by Vaswani et al., replaced recurrence and convolution with attention entirely. A transformer encoder consists of alternating layers of multi-head self-attention and feedforward networks, with residual connections and layer normalization around each. A transformer decoder has the same structure but also includes cross-attention to attend to the encoder's output.

The self-attention mechanism is the heart of the transformer. For each position in the input sequence, self-attention computes a query vector, a key vector, and a value vector. The output for each position is a weighted sum of value vectors, where the weights are determined by the dot products of the query with all keys, normalized by a softmax function. This allows each position to attend to all other positions with a complexity that is quadratic in the sequence length but constant in the number of layers.

Positional encoding addresses the fact that self-attention has no built-in notion of order. Sine and cosine functions of different frequencies are added to the token embeddings before the first attention layer, providing the model with position information. More recent architectures use learned positional embeddings or relative position encodings. RoPE (Rotary Position Encoding) has become particularly popular in recent language models.

The transformer's parallelizability was decisive. Unlike RNNs, which must process sequences sequentially, transformers can process all positions simultaneously during training. This makes them far faster to train on modern GPU hardware, which is optimized for massively parallel computation. A transformer can be trained on the same task in a fraction of the time of an equivalent RNN.

Transformers have been applied far beyond language. Vision Transformer (ViT) processes images by dividing them into patches and treating each patch as a token. AudioSpectrogram Transformer applies transformers to audio. Protein structure prediction with AlphaFold 2 uses transformers. Diffusion models for image generation use transformers as a backbone. The transformer is the universal architecture of modern deep learning in a way that no previous architecture has been.

Training at Scale

The training of large neural networks has become a distinct engineering discipline. Training a frontier model like GPT-4 or Gemini Ultra requires tens of thousands of specialized processors, sophisticated distributed computing systems, carefully curated datasets containing trillions of tokens, and months of wall-clock time. The infrastructure required is more similar to building a large physics experiment than to typical software development.

Distributed training across many devices requires careful handling of communication overhead. The most common parallelism strategies are data parallelism (each device processes a different batch of data and gradients are averaged), model parallelism (different parts of the model reside on different devices), and pipeline parallelism (different layers reside on different devices with micro-batching to keep all devices busy). Frontier models require all three simultaneously, carefully orchestrated.

Mixed-precision training, which uses 16-bit floating point numbers for most computations while maintaining a 32-bit master copy of weights, reduces memory consumption and increases throughput without significant quality loss. Gradient checkpointing trades computation for memory by recomputing intermediate activations during the backward pass rather than storing them. These techniques allow much larger models to be trained on a given amount of hardware.

The data pipeline for large language model training is enormous. Common crawl, a dataset of web pages scraped from the internet, contains hundreds of billions of documents. Processing this into a training dataset involves deduplication, quality filtering, language filtering, and toxic content removal. The quality of training data has a profound effect on model quality. Improvements in data curation have contributed substantially to the capability gains of recent models.

Training stability is a major challenge at scale. Large models can fail to train reliably, experiencing loss spikes where the training loss suddenly increases dramatically, after which recovery may be incomplete. Understanding and preventing training instability is an active area of research. Techniques including careful learning rate scheduling, gradient clipping, weight initialization schemes, and architecture choices all affect stability. Some training runs at the frontier have required restarts after encountering spikes, losing weeks of compute time.

Current Frontiers

The current frontier of neural network research is dominated by several interconnected problems. Scaling laws describe the relationship between model size, dataset size, compute budget, and model performance. Empirically, increasing any of these dimensions tends to improve performance in a predictable way. But scaling becomes increasingly expensive, and understanding where the scaling laws break down is crucial for planning future research.

Mixture of Experts (MoE) architecture addresses the efficiency challenge by activating only a subset of the model's parameters for each input. A large MoE model might have hundreds of billions of total parameters, but each forward pass uses only a small fraction of them. This allows models to have much greater total capacity without proportionally greater computational cost per inference. DeepSeek V3 and Google's Gemini 1.5 Pro both use MoE architectures.

Multimodality is increasingly standard. The most capable models process text, images, audio, and video together. Building models that can learn from and reason about multiple modalities simultaneously requires extending transformer architectures with modality-specific encoders, cross-modal attention mechanisms, and training strategies that handle the different statistical properties of different modalities.

Alignment and interpretability are research areas of growing importance. Neural networks learn through gradient descent from data, without explicit guidance about what concepts they should represent or how they should reason. The internal representations of large models are poorly understood. Mechanistic interpretability research attempts to understand what computations specific circuits in a model perform. This work is crucial both for understanding model capabilities and for identifying potential failure modes.

The efficiency frontier continues to advance. Distillation techniques allow smaller models to learn from larger ones, producing compact models that retain much of the capability of their larger teachers. Quantization reduces the numerical precision of model weights from 32 bits to 8 or 4 bits, dramatically reducing memory requirements with modest quality loss. These techniques are making it possible to run capable models on consumer hardware that would have been impossible to run on frontier hardware just a few years ago.