Time to First Token.

Time to First Token (TTFT) is a critical latency metric that measures how long it takes for a large language model to begin generating its response after receiving a user's request. This metric captures the delay users experience before seeing any output, making it essential for perceived responsiveness in interactive applications. Unlike total generation time, TTFT focuses specifically on the initial delay, which often determines whether an application feels snappy or sluggish to users.

TTFT is primarily dominated by the prefill phase, where the model processes the entire input prompt and builds the initial key-value cache before generating any tokens. During this phase, the model must encode all input tokens, perform attention computations across the full context, and prepare its internal state for generation. The duration depends heavily on input length, model size, hardware capabilities, and system load, with longer prompts requiring proportionally more prefill time.

For user-facing applications, TTFT often matters more than overall throughput because it directly impacts perceived performance and user experience. Applications requiring real-time interaction, such as chatbots or coding assistants, prioritize low TTFT over high tokens-per-second rates. Common optimization strategies include prompt caching, speculative decoding, and batching techniques, though these may involve trade-offs with other performance metrics or increase system complexity.