- Training happens once using thousands of GPUs over months, while inference occurs billions of times daily using smaller hardware per request.
- GPT-4 training reportedly cost over $100M, with Llama 3 70B requiring thousands of A100 GPUs for months on 15 trillion tokens.
- Fine-tuning costs orders of magnitude less than pre-training, enabling organizations to adapt open-weight models without billion-dollar budgets.
- Modern techniques like batching, KV caching, quantization, and speculative decoding reduce inference costs by 5-20× compared to naive implementation.
- A 70B parameter model requires roughly 140GB of GPU memory for weights alone at 16-bit precision during inference.
Two Completely Different Activities
Training and inferenceInferenceThe process of running a trained AI model to generate outputs — what happens when you send a prompt and receive a response.Learn more → are fundamentally different computational activities that happen at different times, at different scales, and require different infrastructure. Training is done once (or occasionally repeated for new versions), by the model's creators, using thousands of GPUs over months. Inference is done billions of times per day, by API users, using a much smaller slice of hardware per request.
Understanding this distinction helps explain LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → economics: the enormous cost of training (GPTGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →-4 was reported to cost over $100M) is amortized across billions of API calls. Inference costs — the marginal cost of generating your response — are what API pricing reflects. They're much smaller per call but enormous in aggregate.
Pre-Training: Learning from the World
Pre-training is where an LLM's capabilities are established. The model is initialized with random weights and trained on trillions of tokens of text using next-tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → prediction. For each training example, the model predicts the next token, the prediction is compared to the actual token (cross-entropyCross-EntropyA loss function that measures the difference between predicted token probabilities and actual tokens, serving as the primary training objective for language models.Learn more → loss), and gradient descentGradient DescentAn optimization algorithm that iteratively adjusts model parameters by moving in the direction opposite to the gradient of the loss function.Learn more → updates the weights to improve future predictions.
Modern pre-training runs consume extraordinary resources: Llama 3 70B was trained on 15 trillion tokens using thousands of A100 GPUs for months. These runs cost tens to hundreds of millions of dollars, which is why frontier pre-training is concentrated among a handful of well-funded labs. The resulting weights encode an enormous compression of human knowledge — these weights are what makes the model valuable.
Fine-Tuning and Alignment
After pre-training, models typically undergo supervised fine-tuningSupervised Fine-TuningSupervised Fine-Tuning (SFT) is the process of training a pre-trained base model on curated instruction-response pairs to teach it specific tasks before preference alignment.Learn more → (SFT) on curated instruction-response pairs, teaching the model to follow directions rather than just complete text. This is followed by reinforcement learning from human feedback (RLHFRLHFReinforcement Learning from Human Feedback — a training technique that uses human preferences to teach AI models to be helpful, honest, and harmless.Learn more →) or similar alignmentAlignmentThe process of ensuring AI models behave according to human values and intentions through training techniques, evaluation methods, and safety measures.Learn more → techniques, which shape the model to be helpful, honest, and to avoid harmful outputs.
Fine-tuningFine-tuningThe process of further training a pre-trained model on a smaller, task-specific dataset to specialize its behavior for a particular use case.Learn more → is far cheaper than pre-training — orders of magnitude less compute. This is why organizations can fine-tune open-weightOpen-WeightA model whose trained weights are publicly available for download, allowing anyone to run, fine-tune, or build on top of it — distinct from fully open-source (which also includes training code and data).Learn more → models for their specific domains without billion-dollar budgets. LoRALoRALow-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique that trains small weight matrices while freezing the base model.Learn more → and other parameter-efficient methods reduce fine-tuning costs further, enabling adaptation on consumer hardware.
Inference: Running the Model
Inference is the forward pass of the model: given an input sequence, compute the next token probability distribution, sample a token, append it to the context, and repeat. The computational cost scales with model size and sequence length. A 70B parameter model requires roughly 140GB of GPU memory for the weights alone (at 16-bit precision).
Modern inference optimization includes batching (processing multiple requests together), KV caching (reusing computed attention states), quantizationQuantizationA compression technique that reduces model size and inference cost by storing weights in lower-precision numerical formats, trading a small amount of accuracy for much lower memory and compute requirements.Learn more → (reducing weight precision to 4-bit or 8-bit), and speculative decodingSpeculative DecodingA technique where a small draft model proposes multiple tokens that a larger model verifies in parallel, maintaining identical output while reducing generation latency.Learn more → (using a small draft model to propose tokens that a large model verifies in parallel). These techniques together reduce inference cost by 5–20× compared to naive implementation.