Architecture
AdamW
A variant of the Adam optimizer that applies weight decay directly to the weights instead of folding it into the gradient as L2 regularization.
In depth
A variant of the Adam optimizer that applies weight decay directly to the weights instead of folding it into the gradient as L2 regularization. Loshchilov and Hutter showed that the two are not equivalent for adaptive optimizers like Adam, and that decoupling them improves generalization and makes the best weight decay value far less dependent on the learning rate. AdamW is the default optimizer for most transformer pretraining and fine-tuning, and ships as torch.optim.AdamW in PyTorch.
Primary source: arxiv.org
Example
Loshchilov and Hutter showed that the two are not equivalent for adaptive optimizers like Adam, and that decoupling them improves generalization and makes the best weight decay value far less dependent on the learning rate.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Architecture.
FAQ
Common questions
What is AdamW?
A variant of the Adam optimizer that applies weight decay directly to the weights instead of folding it into the gradient as L2 regularization.
Why does AdamW matter for AI developers?
AdamW sits in the Architecture part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about AdamW?
Developers Digest publishes tutorials and videos that cover Architecture topics including AdamW. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Architecture
Muon Optimizer
An optimizer for the 2D weight matrices in a neural network's hidden layers that takes the SGD-momentum update and orthogonalizes it with a Newton-Schulz iteration before applying it.
Architecture
Attention Sinks
Attention sinks are the first few tokens of a sequence, which LLMs learn to send large attention scores to even when those tokens carry no special meaning.
Architecture
Autoregressive Model
A model that generates a sequence one element at a time, feeding each output back in as input for the next step.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever