Architecture
Residual Connection
A shortcut that adds a layer's input straight to its output, so the layer only has to learn the change (the residual) instead of a whole new transformation.
In depth
A shortcut that adds a layer's input straight to its output, so the layer only has to learn the change (the residual) instead of a whole new transformation. He et al. introduced it in ResNet in 2015 to make very deep networks trainable, and their ImageNet models went up to 152 layers. The Transformer puts a residual connection around every attention and feed-forward sub-layer, and the running sum of those outputs is often called the residual stream.
Primary source: arxiv.org
Example
In practice, developers reach for Residual Connection when they need the capability described above as part of an AI feature or workflow.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Architecture.
FAQ
Common questions
What is Residual Connection?
A shortcut that adds a layer's input straight to its output, so the layer only has to learn the change (the residual) instead of a whole new transformation.
Why does Residual Connection matter for AI developers?
Residual Connection sits in the Architecture part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about Residual Connection?
Developers Digest publishes tutorials and videos that cover Architecture topics including Residual Connection. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Architecture
Attention Sinks
Attention sinks are the first few tokens of a sequence, which LLMs learn to send large attention scores to even when those tokens carry no special meaning.
Architecture
Backpropagation
The algorithm that computes how much each weight in a neural network contributed to the error, so gradient descent knows how to adjust it.
Architecture
Tensor Parallelism
A model-parallel technique that splits the weight matrices inside each layer across several GPUs, so every GPU computes a slice of every layer and the partial results are combined with collective communication.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever