Architecture
Backpropagation
The algorithm that computes how much each weight in a neural network contributed to the error, so gradient descent knows how to adjust it.
In depth
The algorithm that computes how much each weight in a neural network contributed to the error, so gradient descent knows how to adjust it. It runs the input forward to get a loss, then applies the chain rule backward from the output layer toward the input, reusing intermediate results so every gradient comes out at a small constant multiple of the forward pass cost. Popularized by Rumelhart, Hinton, and Williams in 1986, it is what loss.backward() does in PyTorch and it underpins the training of every modern LLM.
Primary source: nature.com
Example
In practice, developers reach for Backpropagation when they need the capability described above as part of an AI feature or workflow.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Architecture.
FAQ
Common questions
What is Backpropagation?
The algorithm that computes how much each weight in a neural network contributed to the error, so gradient descent knows how to adjust it.
Why does Backpropagation matter for AI developers?
Backpropagation sits in the Architecture part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about Backpropagation?
Developers Digest publishes tutorials and videos that cover Architecture topics including Backpropagation. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Architecture
Tensor Parallelism
A model-parallel technique that splits the weight matrices inside each layer across several GPUs, so every GPU computes a slice of every layer and the partial results are combined with collective communication.
Architecture
Muon Optimizer
An optimizer for the 2D weight matrices in a neural network's hidden layers that takes the SGD-momentum update and orthogonalizes it with a Newton-Schulz iteration before applying it.
Architecture
Residual Connection
A shortcut that adds a layer's input straight to its output, so the layer only has to learn the change (the residual) instead of a whole new transformation.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever