Expert Parallelism

In depth

A distributed training and inference strategy for Mixture of Experts models where different experts live on different GPUs instead of every device holding a full copy. A router picks experts for each token, and an all-to-all collective sends tokens to the devices that host those experts, then gathers the outputs back. Google's GShard paper used this to train a 600B-parameter MoE translation model, noting that experts are too large to replicate so the only viable option is to shard them across devices. Inference engines like vLLM now offer expert parallelism for serving large open-weight MoE models across several GPUs.

Primary source: arxiv.org