RAG & Retrieval
Feed-Forward Network (FFN)
The feed-forward network is the per-token block in every transformer layer that follows attention: two linear projections with a nonlinearity between them, applied to each position independently.
In depth
The feed-forward network is the per-token block in every transformer layer that follows attention: two linear projections with a nonlinearity between them, applied to each position independently. In the original Transformer it expanded a 512-dimension vector to 2,048 and back with a ReLU in between. FFN layers hold most of a model's parameters, which is why Mixture of Experts models replace the single FFN with many expert FFNs and route each token to a few of them.
Primary source: arxiv.org
Example
In practice, developers reach for Feed-Forward Network (FFN) when they need the capability described above as part of an AI feature or workflow.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover RAG & Retrieval.
FAQ
Common questions
What is Feed-Forward Network (FFN)?
The feed-forward network is the per-token block in every transformer layer that follows attention: two linear projections with a nonlinearity between them, applied to each position independently.
Why does Feed-Forward Network (FFN) matter for AI developers?
Feed-Forward Network (FFN) sits in the RAG & Retrieval part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about Feed-Forward Network (FFN)?
Developers Digest publishes tutorials and videos that cover RAG & Retrieval topics including Feed-Forward Network (FFN). Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
RAG & Retrieval
ALiBi
ALiBi (Attention with Linear Biases) is a way to give a transformer position information by penalizing attention scores in proportion to the distance between query and key, instead of adding positional embeddings to the token embeddings.
RAG & Retrieval
Positional Encoding
Positional encoding is the signal a transformer uses to know where each token sits in a sequence, since self-attention on its own treats input as an unordered set.
RAG & Retrieval
Multi-Head Latent Attention (MLA)
Multi-head latent attention is an attention design introduced by DeepSeek that compresses keys and values into a small shared latent vector, so the model caches that latent instead of full per-head keys and values.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever