Tensor Parallelism.

Tensor parallelism is a distributed computing strategy that addresses the memory limitations of deploying large language models by partitioning individual weight matrices across multiple GPUs. Unlike data parallelism, which replicates the entire model on each device, tensor parallelism splits the model itself, allowing each GPU to hold only a portion of the weights while collectively performing the same computation. This approach is essential for serving models with hundreds of billions or trillions of parameters that exceed the memory capacity of even the most powerful individual GPUs.

The technique works by dividing weight matrices along specific dimensions and distributing the resulting tensor slices across available GPUs, with each device computing its portion of matrix multiplications in parallel. Communication between GPUs occurs through high-speed interconnects like NVLink or InfiniBand to synchronize intermediate results and maintain computational correctness. Tensor parallelism can be combined with other parallelization strategies like pipeline parallelism and data parallelism to create hybrid approaches that optimize both memory usage and computational throughput across large GPU clusters.

While tensor parallelism enables deployment of massive models, it introduces communication overhead that can impact inference latency, especially when GPUs are distributed across multiple nodes. The technique requires careful optimization of tensor splitting strategies and communication patterns to minimize bottlenecks, and the benefits diminish when communication costs exceed computational savings. Modern frameworks like Megatron-LM and FasterTransformer implement sophisticated tensor parallelism schemes, but practitioners must balance model size capabilities against the increased infrastructure complexity and potential latency trade-offs.