Mixture of Experts.
A neural network architecture where only a subset of model parameters are activated for each input token, enabling very large models that are surprisingly efficient to run.
Mixture of Experts (MoE) is an architecture that replaces the dense feed-forward layers in a transformer with a collection of 'expert' networks and a routing mechanism. For each token, the router selects a small number of experts (typically 2–8 out of dozens) to process that token. This means a model with 100 billion total parameters might only activate 10 billion for any given input.
MoE models offer a favorable trade-off: they can achieve the capability of a large dense model at the inference cost of a much smaller one. GPT-4 is widely believed to use MoE architecture. Mixtral 8x7B demonstrated that a 47B parameter MoE model could outperform a 70B dense model. Meta's Llama 4 Maverick uses 128 experts.
The main challenges of MoE are training stability, load balancing across experts, and memory footprint. All expert weights must be loaded into memory even though only a fraction are used per token. Despite these challenges, MoE has become the dominant architecture for frontier models that need to balance raw capability with inference efficiency.