Tensor Parallelism

In depth

A model-parallel technique that splits the weight matrices inside each layer across several GPUs, so every GPU computes a slice of every layer and the partial results are combined with collective communication. Megatron-LM popularized it for transformers by partitioning attention heads and MLP matrices with only a few all-reduce operations per layer. Because GPUs synchronize on every layer, it needs a fast interconnect and is usually kept within one server, set in serving engines with options like vLLM's --tensor-parallel-size.

Primary source: arxiv.org