Mixture of Experts (MoE)

In depth

A model architecture that routes each input to a small subset of specialized sub-networks ("experts") rather than activating the entire model. A gating network decides which experts handle each token, so the model can have a massive total parameter count while only using a fraction of them per inference pass. MoE powers models like Mixtral and GPT-4, delivering strong performance at lower compute cost than dense models of equivalent size.