Throughput.

Throughput measures the volume of work an AI system can handle over time, typically expressed as tokens per second or requests per minute. Unlike latency, which focuses on how quickly a single request is processed, throughput captures the system's overall capacity to serve multiple users simultaneously. This metric is crucial for understanding scalability and resource utilization in production AI deployments.

Throughput depends on factors like model size, hardware resources, batching strategies, and parallelization techniques. Systems can achieve high throughput by processing multiple requests in parallel, even if individual requests take longer to complete. The relationship between throughput and latency often involves trade-offs - optimizing for maximum throughput might increase per-request latency, while minimizing latency might reduce overall throughput capacity.

Understanding throughput is essential for capacity planning, cost optimization, and user experience design in AI applications. High-throughput systems can serve more users cost-effectively but may sacrifice individual response times. Organizations must balance throughput requirements with latency expectations based on their specific use cases, whether serving real-time chat applications or batch processing large document collections.