Architecture
Muon Optimizer
An optimizer for the 2D weight matrices in a neural network's hidden layers that takes the SGD-momentum update and orthogonalizes it with a Newton-Schulz iteration before applying it.
In depth
An optimizer for the 2D weight matrices in a neural network's hidden layers that takes the SGD-momentum update and orthogonalizes it with a Newton-Schulz iteration before applying it. The name stands for MomentUm Orthogonalized by Newton-Schulz, and Keller Jordan's write-up credits it with NanoGPT and CIFAR-10 training speed records. Embeddings, output heads, and scalar parameters are still handled by a standard optimizer like AdamW. PyTorch now ships it as torch.optim.Muon, and labs such as Cursor and Thinking Machines use it for large training runs.
Primary source: kellerjordan.github.io
Example
Embeddings, output heads, and scalar parameters are still handled by a standard optimizer like AdamW.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Architecture.
FAQ
Common questions
What is Muon Optimizer?
An optimizer for the 2D weight matrices in a neural network's hidden layers that takes the SGD-momentum update and orthogonalizes it with a Newton-Schulz iteration before applying it.
Why does Muon Optimizer matter for AI developers?
Muon Optimizer sits in the Architecture part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about Muon Optimizer?
Developers Digest publishes tutorials and videos that cover Architecture topics including Muon Optimizer. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Architecture
AdamW
A variant of the Adam optimizer that applies weight decay directly to the weights instead of folding it into the gradient as L2 regularization.
Architecture
Mixed Precision Training
A technique for training neural networks that runs most of the math in lower-precision formats like FP16 or BF16 while keeping critical values in 32-bit.
Architecture
Mechanistic Interpretability
Mechanistic interpretability is the research field that tries to reverse engineer what a neural network computes internally into algorithms humans can understand, the way a programmer might decompile a binary back into readable source code.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever