Data Parallelism

In depth

A distributed training technique that puts a full copy of the model on every GPU and gives each copy a different slice of the training batch. Each replica computes gradients independently, then the replicas exchange and average those gradients every step so all copies stay identical. It is the simplest way to scale training to more hardware, but it only works when the whole model, its gradients, and optimizer state fit on a single device. PyTorch's DistributedDataParallel adds gradient bucketing and overlaps communication with computation, reaching near-linear scaling on 256 GPUs in the original paper.

Primary source: arxiv.org