- MoE activates only a subset of experts per token, enabling 400B parameter models to use just 50B active parameters for inference.
- A small linear layer scores each expert and selects top-k experts, with outputs combined using weighted sums proportional to scores.
- Mixtral 8x7B outperformed Llama 2 70B while being 2× cheaper, and DeepSeek V3 achieves GPT-4o-level performance with 37B active from 671B total parameters.
- All expert weights must be loaded into GPU memory even when inactive, requiring 4× the memory of equivalent dense models.
- MoE models face expert collapse, routing volatility, and load imbalance issues that make training more complex than dense models.
The Core Idea: Conditional Computation
In a standard dense transformerTransformerThe neural network architecture that underpins virtually all modern LLMs, introduced in 2017, built around self-attention mechanisms that process entire sequences in parallel.Learn more →, every parameter is used for every tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more →. A 70B parameter model performs 70B parameter computations per token. Mixture of ExpertsMixture of ExpertsA neural network architecture where only a subset of model parameters are activated for each input token, enabling very large models that are surprisingly efficient to run.Learn more → (MoE) breaks this: instead of one large feed-forward networkFeed-Forward NetworkThe position-wise multilayer perceptron within each transformer block that processes tokens individually after attention, containing most of the model's parameters.Learn more →, each transformer layer has multiple 'expert' feed-forward networks. A router network selects which experts to activate for each token — typically 2 out of 8, 16, or even 128 experts.
The result: a model with 400 billion total parametersParametersThe numerical weights inside a neural network that are learned during training — the 'knowledge' of the model, measured in billions for modern LLMs.Learn more → might only use 50 billion 'active' parameters per token. You get the capability of a large model at the inferenceInferenceThe process of running a trained AI model to generate outputs — what happens when you send a prompt and receive a response.Learn more → cost of a smaller one. This is the key insight that MoE exploits: not all knowledge is needed for all tasks, so routing inputs to specialized sub-networks reduces wasted computation.
How Routing Works
The router is a small linear layer that takes the token representation and produces a score for each expert. The top-kTop-KA sampling parameter that restricts token selection to the k most probable next tokens, discarding the long tail of unlikely options.Learn more → experts by score are selected, and their outputs are combined using a weighted sum (weights proportional to scores). The router is trained jointly with the rest of the model — it learns to send different types of content to different experts.
Load balancing is critical: if all tokens route to the same few experts, the others are wasted capacity. Auxiliary losses during training encourage the router to distribute load evenly across experts. DeepSeek's innovation of auxiliary-loss-free load balancing — using a bias term adjusted dynamically rather than a training loss — is a notable recent improvement.
Notable MoE Models
GPTGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →-4 is widely believed to use MoE architecture, though OpenAI has never confirmed this. Mixtral 8x7B (Mistral) was the first widely adopted open-weightOpen-WeightA model whose trained weights are publicly available for download, allowing anyone to run, fine-tune, or build on top of it — distinct from fully open-source (which also includes training code and data).Learn more → MoE model: 8 experts per layer, 2 active per token, 47B total parameters but 12.9B active — it outperformed Llama 2 70B on most benchmarks while being roughly 2× cheaper to run.
Meta's Llama 4 Maverick uses 128 experts (2 active), with 400B+ total parameters. DeepSeek V3 uses a 671B MoE with 37B active parameters, achieving GPT-4o-level performance at a fraction of the inference cost. Google's Gemini models are also believed to use MoE at scale.
MoE Tradeoffs and Challenges
MoE's benefits come with real costs. Memory: all expert weights must be loaded into GPU memory even though only 2 are active per token. Serving a 400B MoE model requires 4× the GPU memory of a 100B dense model, even if inference compute is similar. This limits MoE's advantage when memory bandwidth rather than compute is the bottleneck.
Training instability: MoE models are harder to train than dense models. Expert collapse (one expert dominates), routing volatility (experts change frequently during training), and load imbalance are all failure modes that require careful engineering. The research community has made substantial progress, but MoE training remains more complex than dense training.