Speculative Decoding.

Speculative decoding is an optimization technique that accelerates large language model inference by leveraging a smaller, faster draft model to propose candidate tokens that are then verified by the target model in parallel. This approach maintains the exact same output quality and distribution as standard autoregressive generation while significantly reducing the wall-clock time required to generate text. The technique exploits the fact that smaller models can often predict the next several tokens correctly, allowing the larger model to process multiple tokens simultaneously rather than one at a time.

The process works by having the draft model generate a sequence of candidate tokens, which the target model then evaluates in parallel using a single forward pass. If the target model agrees with the draft predictions, those tokens are accepted; if not, the process falls back to the target model's prediction and rejects subsequent draft tokens. This verification happens through probability comparison rather than sampling, ensuring that the final output distribution remains mathematically identical to standard generation. The key insight is that modern transformer architectures can efficiently process multiple tokens in parallel during the verification step.

Speculative decoding offers substantial speedups with no quality trade-offs, making it particularly valuable for latency-sensitive applications like real-time chat or interactive coding assistants. The effectiveness depends on how well the draft model can predict the target model's behavior, with better draft models yielding higher acceptance rates and greater speedups. Common misconceptions include thinking this changes the model's output or requires retraining; in reality, it's a pure inference optimization that works with any pre-trained model pair where the draft model has compatible tokenization.