A mixture of experts (MoE) is a neural network architecture that splits a layer into multiple expert subnetworks and uses a learned router to direct each input to a small subset of them. In sparse MoE language models, each token activates only a fraction of total parameters, separating model capacity from per-token compute.
How Does a Mixture of Experts LLM Work?
A mixture of experts LLM works by replacing the standard feed-forward block inside Transformer layers with multiple parallel expert networks controlled by a per-token router. Standard Transformer models alternate between attention layers and dense feed-forward network (FFN) layers. When a language model uses an MoE design, the attention block stays shared across all inputs, while the FFN block becomes a collection of separate expert subnetworks, each typically built as its own FFN.
How the Router and Experts Process Each Token
Every token in an AI model moves through the Transformer layers one step at a time, and the router evaluates each token independently. The router, also called a gating network, is a learned component inside the layer. In typical implementations, the router consists of a small linear layer followed by a softmax function that scores every available expert for the incoming token, paired with a top-k selection step.
Once the router picks the top k experts for that token, only those selected experts compute outputs for it. The model then combines the outputs from the chosen experts, typically by weighting and adding them together, before passing the hidden state to the next Transformer block.
Sparse MoE vs. Dense MoE
Practitioners distinguish between dense MoE and sparse MoE based on how many experts run on each input. In a dense MoE layer, every expert runs on every input. In a sparse mixture of experts layer, only the top-k subset of experts selected by the router runs for a given token. Unless an architecture note explicitly specifies dense routing, modern LLM documentation uses MoE as shorthand for the sparse case.
Active Parameters vs. Total Parameters in an MoE Model
Understanding active parameters vs. total parameters is essential when reading any MoE model card. Because a sparse MoE model stores many expert networks while activating only a few per token, a single parameter count cannot describe both its memory size and its per-token computation.
Total parameters, sometimes called sparse parameters, count every weight across the entire model: all expert FFNs, the routers, the shared attention blocks, and the embedding and output layers. As model builders add more experts to each layer, the total parameter count increases.
Active parameters count only the subset of weights executed during a forward pass for one token. That figure includes the shared layers that run on every token plus the k experts chosen at each MoE layer. Increasing k raises active parameters directly, while keeping k fixed and adding more total experts leaves active parameters nearly unchanged aside from minor router overhead. Forward-pass compute and per-token latency track active parameters closely, whereas the memory required to load the model still tracks total parameters.
Mixture of Experts vs. Dense Model
Comparing a mixture of experts vs. a dense model comes down to how parameter counts map onto per-token computation and system memory. In a dense Transformer, every weight in every feed-forward layer participates in processing every token. Scaling up a dense model’s capacity forces compute per token to grow at the same pace.
A sparse MoE architecture breaks that lockstep relationship. By increasing the total number of experts n while holding the routed count k constant, engineers can expand total parameter capacity while keeping per-token compute roughly tied to k plus the small cost of running the router.
| Dimension | Dense Model | Sparse MoE Model |
|---|---|---|
| Total parameters | One unified set of weights across all layers | Sum of shared layers plus all n expert subnetworks |
| Active parameters per token | Equal to total parameters (100% active) | Subset equal to shared layers plus top-k experts per layer |
| Per-token compute | Scales directly with total parameter count | Scales primarily with active parameters (plus minor router cost) |
| Memory to load the model | Scales with total parameters | Scales with total parameters (all experts usually stay in memory) |
The table highlights a critical rule for model evaluation. When you compare runtime speed or per-token compute efficiency, comparing a sparse MoE model against a dense model of similar active size gives a fair baseline, whereas comparing against a dense model matching its total parameter count compares mismatched workloads.
Where Did Mixture of Experts Come From?
The mixture of experts concept predates modern large language models by more than two decades. Researchers explored conditional routing long before Transformer architectures existed, gradually adapting the idea to massive scale.
In 1991, Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton published Adaptive Mixtures of Local Experts in Neural Computation, introducing a supervised learning framework where a gating network routes different inputs to separate local expert networks.
In 2017, Noam Shazeer and co-authors adapted sparse gating to large language modeling in Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, presented at ICLR 2017. Their work demonstrated that inserting sparsely gated MoE layers lets neural networks scale to far higher parameter counts without proportional increases in per-example computation.
In 2021, William Fedus, Barret Zoph, and Noam Shazeer introduced Switch Transformers. Instead of routing tokens to two or more experts, Switch routing sends each token to k=1 expert. The authors argued that single-expert routing can preserve model quality while lowering router computation and cross-device communication compared to k > 1 routing.
Worked Example: Mixtral 8x7B
Mistral AI’s Mixtral 8x7B release provides a clear, first-party public example of a Mixtral mixture of experts model with fully documented architecture numbers. Announced on December 11, 2023 in the Mixtral of experts blog post and detailed in the accompanying Mixtral of Experts paper, Mixtral 8x7B is a sparse mixture-of-experts decoder-only language model released under the Apache 2.0 license with a 32k token context length.
In Mixtral 8x7B, each Transformer layer’s feed-forward block contains 8 expert parameter groups. For every token at every layer, a learned router chooses 2 of those 8 experts (k=2) and combines their outputs additively.
Mistral AI’s release notes report 46.7B total parameters and 12.9B parameters used per token (the technical paper rounds those figures to 47B total and 13B active). Those numbers illustrate why multiplying 8 by 7B produces a misleading estimate: shared components such as attention blocks are not replicated eight times, so total parameters sit at 46.7B while per-token execution touches only 12.9B.
As an early, well-documented open-weight MoE release, Mixtral 8x7B shows the core mechanics clearly. Later frontier models have also adopted MoE-style sparse feed-forward layers, though commercial providers frequently leave exact expert counts and routing layouts undisclosed.
What Does MoE Mean for Cost and Model Selection?
Mixture of experts architectures alter the balance between per-token inference cost and hardware memory capacity when choosing an AI model. Knowing how those two constraints separate helps teams avoid surprises when moving from model cards to production hosting or API evaluation.
Per-Token Compute vs. Memory Footprint
Because forward-pass compute tracks active parameters rather than total parameters, a sparse MoE model can process each token with compute closer to a smaller dense model. Lower per-token compute often translates into faster per-token execution or lower compute cost per generated token, which matters if you are tracking ways to reduce AI API costs on high-volume workloads.
Memory planning works differently. Serving an MoE model normally requires enough memory and memory bandwidth to hold the entire set of experts simultaneously, because different tokens in a prompt or batch can route to any expert in any layer. Unless a deployment uses parameter offloading, hosting Mixtral 8x7B still requires fitting all 46.7B parameters into memory even though each token activates 12.9B.
Choosing a Fair Comparison Baseline
When I review a model card for a deployment project, I record both numbers side by side: total parameters for memory sizing and active parameters for compute expectations. Matching an MoE model against a dense model of similar active size gives a realistic sense of per-token compute, while total parameter count tells you what hardware tier can actually load the weights.
What Is Mixture of Experts Not?
Clarifying what an MoE model is not prevents common misconceptions about training workflows, external data lookup, and what individual experts actually learn.
Architecture vs. Training and Retrieval
Mixture of experts is a neural network architecture pattern built into the layers of a model. It serves a completely different function from post-training methods such as fine-tuning or distillation, which adapt existing weights or transfer capabilities into a smaller dense model.
Mixture of experts also serves a different role from retrieval-augmented generation (RAG). RAG queries external knowledge bases at runtime to inject relevant context into a prompt, whereas an MoE router simply directs tokens across internal subnetworks inside the model’s fixed weights. Using an MoE model does not give an application live document retrieval.
Do Experts Specialize in Human Topics?
Hearing the term expert often leads people to assume one subnetwork handles Python code, another handles Spanish grammar, and another handles math proofs. Without empirical analysis from a cited paper demonstrating such behavior on a specific checkpoint, you should avoid assuming human-interpretable topic specialization. During training, the learned router partitions inputs mathematically to minimize training loss, and clean human subject labels are not guaranteed.
FAQ
These answers address common practical questions practitioners ask when evaluating sparse mixture of experts language models.
What is an MoE in plain English?
An MoE, or mixture of experts, is a neural network design that replaces one large feed-forward layer with several smaller expert subnetworks and a learned router. For each incoming token, the router selects only a small subset of experts to run, keeping per-token compute lower than the model’s total parameter size.
What is the difference between a dense model and an MoE model?
A dense language model uses 100% of its parameters to process every single token. A sparse MoE language model stores multiple expert networks per layer and routes each token to only the top-k experts. As a result, dense models tie compute directly to total parameters, while sparse MoE models tie compute mainly to active parameters.
Why does an MoE model still require so much memory to load?
Even though each token uses only a fraction of an MoE model’s parameters, different tokens can route to any expert across the network at runtime. Unless a serving framework relies on parameter offloading, the system must keep all total parameters loaded in memory so every expert stays immediately reachable.
How many experts does an MoE LLM activate per token?
The number of active experts per token, denoted as k, depends on the model’s design. Switch Transformer routed each token to exactly one expert (k=1) to minimize routing overhead and communication. Public models such as Mixtral 8x7B select two out of eight experts per layer (k=2) and add their outputs together.
Does mixture of experts replace RAG or fine-tuning?
Mixture of experts does not replace RAG or fine-tuning because each solves a different problem. MoE defines how internal layers route computation inside the model weights. RAG retrieves external documents to ground answers in current data, while fine-tuning updates model behavior for specialized tasks or response formats.
Sources
The architectural definitions, historical milestones, and Mixtral 8x7B figures in this explainer come from the following primary research papers and official release documentation.
- Adaptive Mixtures of Local Experts (Jacobs, Jordan, Nowlan, and Hinton, 1991)
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Shazeer et al., 2017)
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (Fedus, Zoph, and Shazeer, 2021)
- Mixtral of experts (Mistral AI blog, December 11, 2023)
- Mixtral of Experts (Mistral AI technical paper, arXiv:2401.04088)



