Prefill.

Prefill is the first phase of LLM inference that occurs when a model receives a new prompt or conversation. During this phase, the model processes all input tokens simultaneously in parallel, computing attention weights and building the key-value (KV) cache that will be used during subsequent token generation. This parallel processing makes prefill computationally intensive but relatively fast per token, as modern hardware like GPUs excel at parallel operations. The prefill phase is essential for establishing the context and attention patterns that guide the model's understanding of the input.

Unlike the decode phase that follows, prefill can leverage the full parallelization capabilities of transformer architectures since all input tokens are known upfront. The model computes attention scores between every token pair in the input, stores the resulting key and value representations in the KV cache, and prepares the initial hidden states for generation. The computational complexity scales quadratically with input length due to the attention mechanism, making longer prompts significantly more expensive during prefill. The phase concludes when the model has processed the entire input and is ready to predict the first output token.

Prefill latency directly impacts time-to-first-token (TTFT), a critical user experience metric, especially for interactive applications. While prefill is typically faster per token than decode due to parallelization, very long prompts can still create noticeable delays. Optimizations like chunked prefill, where long inputs are processed in segments, help balance memory usage and latency. Understanding prefill behavior is crucial for applications that frequently send long contexts, as the quadratic scaling means doubling prompt length roughly quadruples prefill computational cost, making context management strategies essential for cost-effective deployment.