FSDP (Fully Sharded Data Parallel)

In depth

A PyTorch training method that splits model parameters, gradients, and optimizer state across GPUs instead of copying them all onto every device as plain data parallelism does. Each GPU gathers the full parameters for one layer group only when it needs them for the forward or backward pass, then frees them, so the memory per device drops roughly with the number of GPUs. Meta's FSDP paper reports throughput comparable to DistributedDataParallel while training much larger models with near-linear scaling. It is a common choice for full fine-tuning of models too large for one GPU.

Primary source: arxiv.org