Pipeline Parallelism

In depth

A model-parallel technique that splits a network by layers into sequential stages on different accelerators, passing activations from one stage to the next. To keep stages from sitting idle, GPipe splits each batch into micro-batches so several stages work at once, and reported almost linear speedup as a model was partitioned across more devices. It communicates less often than tensor parallelism, so it is often used across servers, at the cost of pipeline bubbles where some stages wait.

Primary source: arxiv.org