Large Language Models: The Definitive Guide

What Is a Large Language Model

A large language model (LLM) is a neural network trained on massive amounts of text to predict what words come next in a sequence. That simple objective, predicting the next token, turns out to be a remarkably powerful learning signal. To predict what word comes next, the model must learn grammar, facts, reasoning patterns, and much of what we might call knowledge about the world.

The 'large' in LLM refers to two things: the number of parameters (the adjustable weights in the neural network) and the size of the training dataset. Modern frontier models have hundreds of billions of parameters and are trained on trillions of tokens of text. The sheer scale of training data and model size produces capabilities that smaller models do not have.

LLMs are autoregressive models. They generate text one token at a time, with each token conditioned on all previous tokens. A token is roughly four characters or three-quarters of a word in English. The model maintains no persistent state between generations. Each new generation starts from the beginning of the conversation context and runs the entire forward pass again. This is one of the reasons LLMs are computationally intensive at inference time.

The transformer architecture, described in the 2017 paper 'Attention Is All You Need,' is the foundation of virtually all modern LLMs. The self-attention mechanism allows the model to process relationships between all tokens in its context window simultaneously, regardless of how far apart they are in the sequence. This is what enables LLMs to take a question and a long document, understand how they relate, and answer the question accurately.

LLMs are general-purpose models. Unlike specialized systems trained for a single task, a well-trained LLM can write code, translate languages, answer factual questions, compose poetry, analyze legal documents, and engage in multi-step reasoning, all without any task-specific modification. This generality is what makes them so remarkable and so potentially valuable.

Pre-Training: Learning from the Internet

Pre-training is the first and most resource-intensive phase of building an LLM. The model is trained on a massive corpus of text, typically hundreds of billions to trillions of words drawn from the internet, books, scientific papers, code repositories, and other sources. The training objective is simple: given the preceding tokens, predict the next token. The model is repeatedly shown text, generates predictions, compares them to the actual next tokens, and updates its weights to make better predictions.

The training corpus is not simply a raw dump of everything on the internet. Quality filtering removes spam, machine-generated text, adult content, and other undesirable material. Deduplication removes repeated content, which is important because the model should not memorize specific documents but rather learn generalizable patterns. Language filtering ensures appropriate language balance. The composition of the training corpus has a profound effect on what the model knows and how it reasons.

Training a frontier LLM takes months on a cluster of thousands of specialized GPUs. The total compute required is measured in floating-point operations, typically in the range of tens to hundreds of zettaflops for frontier models. The cost of a single training run is tens of millions of dollars. This creates a significant barrier to entry and concentrates frontier model development in a small number of well-capitalized organizations.

Scaling laws describe how model performance improves with scale. Empirically, increasing the model size (number of parameters) and the amount of training data, in proportion, leads to predictable improvements in performance on language benchmarks. The Chinchilla scaling laws, published by DeepMind in 2022, showed that previous large models had been undertrained: they had more parameters than was optimal for their compute budget, and would have performed better if they had been trained on more data with fewer parameters at the same total compute cost.

Pre-training produces a base model, sometimes called a foundation model. A base model is very good at completing text in the style of whatever it was trained on. It is not yet useful as an assistant. Given a question, a base model might respond with more questions, or with a list of similar questions from its training data, rather than answering. The next phases of development transform the base model into a usable assistant.

Fine-Tuning and RLHF

Supervised fine-tuning (SFT) is the first step in transforming a base model into an assistant. Human contractors write examples of ideal assistant behavior: questions with high-quality answers, instructions with complete and helpful responses. The model is trained on these examples to imitate the demonstrated behavior. SFT significantly changes the model's output distribution, making it much more likely to produce helpful, direct responses.

Reinforcement Learning from Human Feedback (RLHF) goes further. Human raters compare pairs of model outputs and indicate which they prefer. These preference judgments are used to train a reward model, a separate neural network that predicts which outputs humans would prefer. The language model is then fine-tuned using reinforcement learning to maximize the reward model's score.

RLHF was the key innovation that transformed GPT-3, a powerful but hard-to-use base model, into InstructGPT and ultimately ChatGPT. The paper describing InstructGPT showed that a 1.3 billion parameter model fine-tuned with RLHF was preferred by human raters over a 175 billion parameter model fine-tuned without RLHF on many tasks. Alignment through human feedback was more important than raw model scale.

Constitutional AI, developed by Anthropic, is an alternative alignment approach. Rather than relying on human preference judgments for every training signal, the model is trained using AI-generated feedback based on a set of explicit principles (the 'constitution'). The AI critiques and revises its own outputs according to these principles. This reduces the labor required for human feedback while maintaining alignment with human values.

Direct Preference Optimization (DPO), introduced in 2023, offers a simpler alternative to RLHF. Rather than training a separate reward model, DPO directly optimizes the language model's policy to increase the probability of preferred responses and decrease the probability of dispreferred ones. It achieves comparable alignment results with a simpler training pipeline. Most frontier models now use some variant of DPO or related techniques.

Architecture: Tokens, Embeddings, and Layers

Text is converted into tokens by a tokenizer before being fed to an LLM. A tokenizer breaks text into subword units using algorithms like Byte-Pair Encoding (BPE) or SentencePiece. Common words are often a single token. Rare words are split into multiple tokens. The tokenizer's vocabulary is fixed at training time, typically containing 30,000 to 100,000 tokens. The choice of tokenizer affects how efficiently different languages and domains are represented.

Each token is mapped to an embedding vector, a high-dimensional numerical representation. Embeddings capture semantic relationships: words with similar meanings have similar embeddings. The embedding dimension of frontier models is typically in the range of 4,000 to 12,000 dimensions. The embedding layer is learned during training and encodes much of the model's basic knowledge about language.

The transformer layers process the sequence of token embeddings. Each layer consists of a multi-head self-attention block and a feedforward network, with residual connections and layer normalization. The self-attention block allows each token to attend to all other tokens in the context. The feedforward network applies a learned transformation to each token independently. Information is progressively integrated and transformed through the layers.

The context window is the maximum number of tokens the model can process in a single forward pass. Early LLMs had context windows of 2,048 tokens. Recent frontier models have context windows of 128,000 (128K) to 1,000,000 (1M) tokens. A larger context window allows the model to work with longer documents, maintain longer conversation histories, and process more code at once. Extending context windows is technically challenging because the quadratic complexity of self-attention becomes prohibitive at very long sequences.

After the final transformer layer, a linear projection maps the output embeddings to a vocabulary-sized vector, and a softmax function converts these to probabilities over all tokens in the vocabulary. The token with the highest probability is selected as the next token, though in practice a sampling strategy with a temperature parameter is used to introduce appropriate randomness. Temperature controls the sharpness of the probability distribution: low temperature makes the model more deterministic, high temperature makes it more varied.

Emergent Capabilities

One of the most striking and controversial aspects of large language models is the emergence of capabilities that were not explicitly trained. As models scale, new abilities appear suddenly, qualitatively changing what the model can do. These are called emergent capabilities. Examples include multi-step arithmetic, chain-of-thought reasoning, in-context learning from a few examples, and the ability to follow complex instructions.

In-context learning is perhaps the clearest example of emergence. Given a few examples of a task in the prompt, a large model can generalize to new instances of that task without any weight updates. A small model given the same examples cannot. This capability appears abruptly at around 100 billion parameters and improves further with scale. It is not clear from first principles why training on text prediction should produce this capability. It simply emerges.

Chain-of-thought reasoning, where the model reasons through a problem step by step before giving a final answer, significantly improves performance on multi-step reasoning tasks. Smaller models do not benefit from chain-of-thought prompting. For large models, it can be transformative, increasing accuracy on math word problems and logical reasoning by substantial margins. The model is using its own generated text as an external workspace for computation.

The concept of emergence is debated. Some researchers argue that many apparent emergences are artifacts of how capabilities are measured. If you measure performance on a binary correct/incorrect metric, improvement can appear suddenly even when the underlying model probability improves smoothly. Other researchers argue that there are genuine phase transitions in model capability as scale increases. The debate has not been fully resolved.

Emergent capabilities are important practically because they mean that evaluating a model at one scale gives limited information about what a larger model will be able to do. Capabilities that seem absent at one scale may appear suddenly at larger scales. This makes it difficult to predict what frontier-scale models will be able to do before they are trained. It is one of the factors that makes the trajectory of AI development hard to forecast.

Hallucination: Why LLMs Make Things Up

Hallucination is the phenomenon where an LLM generates text that is factually incorrect but stated with apparent confidence. The model might cite a paper that does not exist, attribute a quote to the wrong person, describe events that never happened, or invent technical specifications for a product. Hallucination is one of the most significant reliability challenges facing LLMs in high-stakes applications.

The fundamental cause of hallucination is that LLMs are trained to generate plausible text, not necessarily true text. The training objective is next-token prediction: predict what word comes next in a sequence drawn from the training corpus. The training corpus itself contains misinformation, inconsistencies, and biases. The model learns to generate text that looks like the training corpus, including plausible-sounding but incorrect claims.

The confidence problem compounds this. LLMs do not have direct access to their own uncertainty. A language model assigns some probability to every possible next token, but the probability mass for incorrect tokens is not zero. When a model encounters a question about something it has seen rarely or inconsistently in training, it may generate a confident-sounding answer that happens to be wrong, because it lacks any mechanism for detecting its own knowledge gaps.

Several techniques reduce hallucination. Retrieval-Augmented Generation (RAG) provides the model with relevant passages from a knowledge base before asking it to answer a question. This grounds the generation in source material that can be verified. RLHF training can be tuned to reward hedging and uncertainty expression. Chain-of-thought prompting can expose reasoning errors that would otherwise be hidden in a direct answer. Constitutional AI explicitly trains models to acknowledge uncertainty.

Hallucination rates vary significantly across models and tasks. Factual question answering about well-documented topics in the training data has low hallucination rates. Questions about obscure topics, recent events, or technical details at the edge of the training distribution have higher hallucination rates. Applying LLMs to high-stakes tasks like medical diagnosis, legal analysis, or financial advice requires robust hallucination mitigation strategies and should always involve human verification.

Context Windows and Memory

The context window is the amount of text an LLM can process in a single forward pass. Everything the model knows about the current conversation or task must fit within this window. Early GPT-3 had a context window of 4,096 tokens, roughly 3,000 words. Claude 3.5 has a 200,000-token context window. Llama 4 Scout has a 10 million-token context window. The expansion of context windows has been one of the most significant practical improvements in LLMs over recent years.

A larger context window enables longer documents to be analyzed, longer conversation histories to be maintained, larger codebases to be processed, and more examples to be included in the prompt for in-context learning. Many practical limitations of early LLMs, such as forgetting earlier parts of a conversation or being unable to answer questions about long documents, were context window limitations. Expanding the window addresses these limitations directly.

But context windows are not free. The quadratic complexity of self-attention means that processing a context twice as long requires four times as much computation. Techniques like Grouped Query Attention (GQA) and Sliding Window Attention reduce this computational cost for very long contexts. But there are fundamental limits to how efficiently long-context processing can be done with the standard transformer architecture.

The 'lost in the middle' problem is a known issue with long contexts. Models tend to attend more strongly to tokens at the beginning and end of the context window. Information buried in the middle of a very long context is attended to less reliably. This means that while a model may technically have a 200K token context window, performance may degrade for information that is many thousands of tokens from the beginning or end of the context.

LLMs have no persistent memory between conversations. Each new conversation starts with an empty context. This means the model does not remember previous interactions with a user unless a summary or the full previous conversation is included in the current context. External memory systems, where important information is stored in a vector database and retrieved into the context as needed, are the standard approach to giving LLMs something like long-term memory.

Multimodality: Beyond Text

The most capable current LLMs are not limited to text. They can process images, audio, and video, and many can generate images and audio as well as text. These multimodal models represent an extension of the transformer architecture to handle different types of input and output, each requiring its own encoding and decoding machinery.

Vision language models (VLMs) process images by dividing them into patches and encoding each patch as a token. These visual tokens are then combined with text tokens in the same transformer. The model can attend across text and images simultaneously, enabling tasks like describing an image, answering questions about an image, or reading text in an image. GPT-4V, Claude 3, and Gemini are all vision language models.

Audio processing follows a similar pattern. Speech is converted to a spectrogram, a two-dimensional representation of how energy is distributed across frequencies over time. Transformers trained on spectrograms can perform automatic speech recognition, speaker identification, and sound classification. OpenAI's Whisper is an example of a highly capable speech recognition model based on this approach.

Generating images from text descriptions requires different architectures. Diffusion models, which learn to reverse a process of progressively adding noise to images, have become the dominant approach for image generation. They are often combined with a text encoder based on a transformer, allowing text prompts to guide the denoising process. DALL-E 3 and Midjourney are based on this approach.

Native multimodality, where a single model architecture processes all modalities together during training, produces better models than combining separately trained models. When a model learns from text and images jointly, it can develop representations that capture multimodal relationships, such as understanding that the word 'red' refers to a visual property. GPT-4o and Gemini Ultra are natively multimodal, trained on text and images simultaneously rather than as separate systems joined after training.

How to Compare Models

Evaluating LLMs is harder than it might seem. Academic benchmarks measure specific, narrowly defined capabilities. A model that tops the leaderboard on MMLU (broad factual knowledge) or HumanEval (Python code generation) may underperform another model on your specific use case. The most important evaluations are the ones you run on your own tasks, with your own prompts, measuring what matters for your application.

Standardized benchmarks provide a useful starting point. MMLU tests knowledge across 57 subjects from elementary mathematics to professional law and medicine. GPQA tests graduate-level science reasoning. HumanEval and SWE-Bench test coding ability. HellaSwag tests commonsense reasoning. MATH and AIME test mathematical problem-solving. Models that do well across multiple benchmarks are generally more capable.

Human preference evaluations are an important complement to benchmark scores. The LMSYS Chatbot Arena asks humans to compare pairs of model outputs without knowing which model produced each, then uses these comparisons to produce an Elo rating. This captures aspects of quality that are difficult to measure with automated benchmarks, such as tone, naturalness, and helpfulness. Models that top the Elo leaderboard are generally considered the most pleasant to interact with.

Speed and cost matter enormously in practice. A model that is 10% more capable but costs 5x as much and has 3x the latency may be the wrong choice for most applications. Measuring tokens per second, time to first token, and cost per thousand tokens on your specific usage patterns is as important as measuring quality. The model you choose is a product of capability, cost, and speed together.

Running the same prompt across multiple models simultaneously is the most direct way to compare them. Tools like Keimodel let you do exactly that: submit a single prompt and see every model's response side by side. The differences between models that look similar on paper often become immediately apparent when you compare their actual outputs on your actual tasks. No benchmark captures this kind of task-specific comparison.

The Future of LLMs

The trajectory of LLM development points toward systems that are more capable, more efficient, and more integrated into workflows. Reasoning capabilities, where models spend more compute at inference time to think through difficult problems, are improving rapidly. Models that can engage in hours of extended reasoning on complex problems, rather than seconds of fast pattern matching, represent a qualitative change in capability.

Agent systems, where LLMs use tools and take sequences of actions to complete long-horizon tasks, are the next major application area. An LLM that can browse the web, write and execute code, manage files, send emails, and coordinate with other AI systems represents a fundamentally different kind of tool than a chatbot. Building reliable, safe, and capable agents is the central engineering challenge of the next several years.

Multimodal reasoning, combining language, vision, audio, and other modalities in integrated reasoning chains, will become standard. A model that can watch a video, read a document, examine a spreadsheet, and synthesize insights across all three modalities is dramatically more useful than a model limited to text. The infrastructure for this kind of multimodal reasoning is being built now.

Efficiency improvements will continue to make frontier capabilities more accessible. Distillation produces compact models that inherit much of the capability of much larger models. Quantization makes large models run on consumer hardware. Speculative decoding speeds up inference without changing outputs. These techniques ensure that the gains achieved at the frontier propagate down to smaller, cheaper, more widely deployed systems.

The longer-term trajectory raises more fundamental questions. If LLMs continue to improve at their current rate, how far do they go? Artificial general intelligence, broadly defined as AI that can perform any cognitive task a human can perform, is explicitly the goal of several major labs. Whether current LLM architectures can achieve AGI through scaling, or whether fundamentally new ideas are needed, is the most important open question in AI research.