In-Context Learning.
The ability of an LLM to learn from examples provided directly in the prompt, without any weight updates or training — just from seeing patterns in context.
In-context learning (ICL) is one of the most striking emergent properties of large language models. Rather than requiring task-specific training, LLMs can adapt their behavior based on examples provided in the prompt itself. Show a model three examples of translating English to French, and it will correctly translate the fourth without any fine-tuning.
ICL comes in several varieties: zero-shot (no examples, just instructions), one-shot (one example), and few-shot (several examples). GPT-3's strong few-shot learning ability was a major reason for its impact in 2020 — it showed that a single model could flexibly perform many tasks without modification.
The mechanism behind ICL is still debated among researchers. Some evidence suggests models perform a form of implicit gradient descent in their forward pass. Others argue they pattern-match to similar sequences seen during training. Practically, the main limitation is context window size: you can only fit so many examples in a single prompt.
Related terms.
- Prompt EngineeringThe practice of crafting inputs, instructions, examples, and formatting to guide language models toward producing better, more accurate outputs.
- Scaling LawsEmpirical power-law relationships showing how model performance improves predictably with increases in model size, training data, and compute resources.