Chain of Thought Prompting: Teaching Models to Reason

Chain-of-thought prompting dramatically improves LLM performance on complex tasks by encouraging models to show their reasoning before answering.

Key takeaways
  • 2022 paper showed adding 'Let's think step by step' improved math problem accuracy from 18% to 79%.
  • Generating intermediate tokens allocates additional compute to hard problems, creating context for subsequent reasoning steps.
  • Chain-of-thought makes reasoning errors visible and correctable, unlike direct-answer prompts that hide failure points.
  • OpenAI o1, o3, and Claude 3.7 Sonnet automatically generate thousands of internal thinking tokens before responding.
  • Reasoning models outperform non-reasoning counterparts but require longer latency and higher per-request costs.

The Chain-of-Thought Breakthrough

In 2022, Google researchers published 'Chain-of-Thought Prompting Elicits Reasoning in Large Language Models', demonstrating that simply asking models to show intermediate reasoning steps dramatically improved accuracy on math, logic, and common-sense reasoning tasks. The key insight: language models can effectively use generated text as a scratchpad for computation.

The paper showed that on multi-step math problems, a standard promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more → asking for a direct answer achieved around 18% accuracy on a benchmarkBenchmarkA standardized test or set of tasks used to evaluate and compare the capabilities of different AI models on a common scale.Learn more →. Adding 'Let's think step by step' pushed accuracy to 79%. This simple trick — requiring intermediate reasoning — unlocked reasoning capabilities that were already present in the model but not utilized without the explicit prompt.

Why Chain of Thought Works

Chain-of-thought works because generating intermediate tokens allocates additional compute to hard problems. Each reasoning step creates new context that subsequent steps can build on, allowing the model to solve problems that are too complex to resolve in a single forward pass. It's analogous to writing out your work on paper — the act of articulation catches errors and clarifies thinking.

CoT also makes errors visible and correctable. In a direct-answer prompt, a wrong answer gives no information about where reasoning failed. In a CoT response, you can often identify the exact step where the model went astray — which is useful both for debugging prompts and for training better models.

Chain of Thought in Modern Reasoning Models

Modern reasoning models like OpenAI o1, o3, and Anthropic's Claude 3.7 Sonnet with extended thinking have internalized chain-of-thought as a core capability trained via reinforcement learning. Rather than requiring a special prompt, these models automatically generate extended internal reasoning before producing a final answer — often spending thousands of tokens 'thinking' before responding.

This represents a shift from CoT as a prompting technique to CoT as a model capability. Reasoning models consistently outperform their non-reasoning counterparts on hard math, competition programming, and complex analysis tasks. The tradeoff is speed and cost: more thinking tokens mean longer latencyLatencyThe delay between sending a request to an LLM and receiving the first token of the response, often measured as Time to First Token (TTFT).Learn more → and higher per-request costs. The research community is actively exploring how to make reasoning more efficient.

Read next
9 min read

Reasoning Models and Chain of Thought: AI That Thinks

How reasoning models work, why they're so much better at hard problems, the key models in the space, and when to use them over standard LLMs.

5 min read

Zero-Shot Prompting: Getting Results Without Examples

Zero-shot prompting asks an LLM to perform a task with no examples — relying entirely on the model's pretrained knowledge and instruction-following ability.

12 min read

Prompt Engineering: The Complete Guide

Master the art and science of writing effective prompts — from basic techniques to advanced methods like chain-of-thought, few-shot learning, and structured output generation.

9 min read

RLHF: How AI Models Learn to Be Helpful

Reinforcement Learning from Human Feedback — the training technique behind ChatGPT and Claude that shaped modern AI assistants to be helpful, harmless, and honest.