- LLM performance improves smoothly and predictably with model size, dataset size, and compute following a mathematical power law.
- Optimal training requires approximately 20 tokens per parameter, meaning a 70B model needs 1.4 trillion tokens of training data.
- Certain capabilities like few-shot learning and chain of thought reasoning appear discontinuously in larger models without explicit training.
- Reasoning models allocate more tokens to 'thinking' for hard problems, extracting more capability from the same model weights at inference.
- The scaling laws show no bottom floor for capabilities that matter, driving the race to build larger compute clusters.
The Discovery That Changed Everything
In 2020, OpenAI published 'Scaling LawsScaling LawsEmpirical power-law relationships showing how model performance improves predictably with increases in model size, training data, and compute resources.Learn more → for Neural Language Models,' demonstrating that LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → performance improves smoothly and predictably with model size, dataset size, and compute. The relationship is a power law: doubling compute improves performance by a consistent multiplicative factor. This meant AI progress could be planned and projected like an engineering problem — not hoped for like a research breakthrough.
The Chinchilla scaling laws (DeepMind, 2022) refined this: the original OpenAI findings trained large models on insufficient data. Chinchilla showed that optimal training requires approximately 20 tokens per parameter — meaning a 70B model needs 1.4 trillion tokens of training data. This insight shifted the field toward training smaller models on more data.
The Three Scaling Axes
Performance scales smoothly with three quantities: N (number of parametersParametersThe numerical weights inside a neural network that are learned during training — the 'knowledge' of the model, measured in billions for modern LLMs.Learn more →), D (dataset size in tokens), and C (compute, measured in FLOPsFLOPsFloating-point operations per second, a measure of computational throughput used to quantify the processing power required for training and running AI models.Learn more →). For a fixed compute budget, there's an optimal allocation between N and D — the Chinchilla optimal frontier. Frontier labs use these scaling laws to decide how large to make their models and how long to train them before deploying.
Importantly, the scaling laws have no bottom floor visible yet for the capabilities that matter — more compute consistently produces better models. This is why the major labs are in a race to build larger and larger compute clusters, and why AI capability improvements have been so rapid and sustained.
Emergent Capabilities at Scale
The scaling laws predict smooth continuous improvements in perplexityPerplexityA statistical measure of how well a language model predicts text — lower perplexity means the model finds the text less surprising and is a better predictor.Learn more →. But certain capabilities appear discontinuously — they're absent in smaller models and suddenly present in larger ones. These emergent capabilities include Few-ShotFew-ShotA prompting technique where you include a small number of input-output examples in the prompt to demonstrate the desired task format and behavior to the model.Learn more → learning, Chain of ThoughtChain of ThoughtA prompting technique that encourages an LLM to reason step by step before giving a final answer, dramatically improving performance on complex tasks.Learn more → reasoning, arithmetic, translation, and code generation. They appeared without explicit training on these tasks. A Foundation ModelFoundation ModelA large AI model trained on broad data at massive scale that can be adapted to a wide variety of downstream tasks, forming the 'foundation' for specialized applications.Learn more → large enough to exhibit these emergent capabilities becomes far more useful as an Instruct ModelInstruct ModelA base language model that has been fine-tuned to follow instructions from users, as opposed to just completing text in the style of its training data.Learn more → after fine-tuningFine-tuningThe process of further training a pre-trained model on a smaller, task-specific dataset to specialize its behavior for a particular use case.Learn more → and alignmentAlignmentThe process of ensuring AI models behave according to human values and intentions through training techniques, evaluation methods, and safety measures.Learn more →.
The emergence of capabilities at scale is a key reason why larger models are not just 'slightly better' at existing tasks — they can do things qualitatively different from what smaller models can do. This unpredictability (which capabilities will emerge and when) makes AI development partially like engineering and partially like discovery.
Test-Time Compute Scaling
The latest scaling frontier is test-time computeTest-Time ComputeUsing additional computation during inference to improve model outputs through techniques like extended reasoning, multiple sampling, or search algorithms.Learn more →: rather than only scaling training, also scale how much compute is used at inferenceInferenceThe process of running a trained AI model to generate outputs — what happens when you send a prompt and receive a response.Learn more →. Reasoning models (o3, DeepSeek R1) allocate more tokens to 'thinking' for hard problems, extracting more capability from the same model weights. The scaling law analog: performance continues to improve as you give the model more tokens to think.
This opens a new dimension of AI improvement that doesn't require building ever-larger training runs. A smaller model with extended reasoning may outperform a larger model without it on sufficiently hard tasks. Adaptive compute allocation — spending more thinking tokens on hard problems than easy ones — is the leading edge of current scaling research.