Tokens per Second.

Tokens per second (TPS) quantifies the rate at which a large language model generates output tokens during text generation, making it the fundamental metric for measuring LLM throughput performance. This measurement captures how many discrete text units—whether words, subwords, or characters depending on the tokenization scheme—a model can produce in one second of processing time. TPS directly impacts user experience in real-time applications like chatbots, coding assistants, and content generation tools, where faster generation means more responsive interactions.

The measurement occurs during the inference phase when models decode tokens sequentially, with each new token depending on all previously generated tokens in the sequence. TPS varies significantly based on factors including model size, hardware specifications, batch size, sequence length, and optimization techniques like quantization or speculative decoding. Unlike latency metrics that measure delay, TPS focuses purely on sustained generation speed once the model begins producing output, making it distinct from time-to-first-token measurements that capture initial response delays.

Higher TPS enables better user experiences and more cost-effective serving at scale, but optimizing for speed often involves trade-offs with model quality, accuracy, or resource requirements. Organizations frequently balance TPS against other metrics like cost per token, model capability, and hardware constraints when selecting models for production deployment. Common misconceptions include conflating TPS with overall model performance or assuming that higher TPS always translates to better user experience, when factors like response quality and contextual accuracy remain equally important for most applications.