Learn · LLM inference

Mixture of experts: 117 B parameters, 5 B per token

Updated 10 Nov 2026 · 7 min read

In every transformer layer, after attention, each token goes through an MLP (see Tokens and transformers). In a dense model that MLP is one big block that every token uses. In a mixture-of-experts (MoE) model it is many smaller blocks, the experts, and each token uses only a few. gpt-oss-120b, the running example on these pages, is an MoE model.

The idea

Picture a clinic with 128 specialists and a receptionist. Each patient (a token) is sent to the 4 specialists the receptionist thinks fit best, and the patient’s notes are a blend of their 4 reports, weighted by how confident the receptionist was. The clinic can employ a huge staff (lots of knowledge stored) while each patient costs only 4 consultations (little compute).

  • The router is the receptionist: it answers “which experts suit this token?”
  • The top-4 cut answers “how many do we pay for?” The other 124 do no work for this token.
  • The weights answer “how much does each opinion count?”
  • The weighted sum is the blended report, added to the token’s vector.

Where the analogy breaks: experts are not topic specialists (no “code expert” or “French expert”); routing is learned per token and per layer, so the same word visits a different 4 in each of the 36 layers. And all 128 specialists must be on site at all times, because any token may need any of them: MoE saves compute, not memory.

Experts per layer
128
4 used per token
One expert
24.9 M
parameters (3 × 2,880 × 2,880)
Stored
116.8 B
parameters, all must be loaded
Used per token
5.1 B
~9 GFLOP of compute
One MoE layer, one token
128 EXPERTS (SWIGLU MLPS)Token vectornormed, 2,880Softmax over the 4 kept scores gives the mixing weights.Router2,880 × 128 → scores, keep top4Expert 74score 2.3 · weight 0.31Expert 47score 2.14 · weight 0.26Expert 127score 2 · weight 0.23Expert 101score 1.9 · weight 0.21124 othersskipped for this tokenWeighted sumadded to residual stream
  1. RouteThe token’s normed hidden state (2,880 numbers) is multiplied by the router matrix: 128 scores, one per expert.
  2. Top 4Keep the 4 highest scores; softmax over those 4 gives weights that sum to 1. The 124 other experts are skipped.
  3. RunEach chosen expert is a full SwiGLU MLP on the same input vector. They are independent, so a kernel runs them side by side.
  4. CombineOutput = Σ weight × expert output, added to the residual stream like any MLP output.
gpt-oss-120b: 128 experts per layer, top-4 routing. Expert numbers and weights are the worked example's below. Another token, or the same token in the next layer, may pick a completely different 4.

Worked example: routing one token

Real gpt-oss layout (128 experts, top 4), but a toy hidden size of 4 instead of 2,880. The router and expert weights come from a seeded random generator; every other number is computed from them at build time.

One token through one MoE layer

1 · The token's vector arrives

x (after RMSNorm)
d1d2d3d4
token1.20−0.400.800.60
×
Router Wr, columns for experts 74 and 47 (of 128)
e74e47
d10.90.9
d2−0.9−0.1
d30.40.9
d40.90.5

The router is a single small matrix: 4 × 128 here, 2,880 × 128 in gpt-oss (0.37 M parameters per layer, tiny next to one expert’s 24.9 M). Each column belongs to one expert. Expert 74’s score: 1.2·0.9 + −0.4·−0.9 + 0.8·0.4 + 0.6·0.9 = 2.3.

2 · 128 scores, one per expert

Router scores x · Wr (row = experts 0–15, 16–31, …)
+0+1+2+3+4+5+6+7+8+9+10+11+12+13+14+15
0–15−0.7−0.70.70.10.6−0.5−1.2−0.70.81.5−0.4−0.10.40.0−1.10.4
16–31−0.60.7−0.71.60.40.6−1.0−0.3−0.3−0.70.00.6−0.80.1−0.10.3
32–471.40.90.3−0.30.3−1.30.0−1.10.3−2.1−1.60.4−0.1−1.6−1.22.1
48–630.60.7−0.60.2−1.4−0.70.1−0.4−0.10.5−1.6−0.90.40.60.90.4
64–79−0.20.1−1.70.60.91.70.90.60.51.32.30.80.2−1.30.4−1.6
80–95−0.1−0.60.81.4−0.30.8−0.90.1−0.20.1−0.8−0.2−2.7−1.21.71.3
96–1110.7−1.60.61.20.31.90.51.41.70.30.60.2−1.61.2−0.41.0
112–127−0.9−0.2−0.10.00.5−0.1−0.21.70.30.00.6−0.2−1.20.00.42.0

Blue is high, red low. The 4 outlined cells are the highest: experts 74, 47, 127, 101. Nothing makes neighbours special; expert numbers are just column indices.

3 · Keep the top 4, softmax over them

Top-4 scores
score
expert 742.30
expert 472.14
expert 1272.00
expert 1011.90
softmax →
Weights (sum to 1)
weight
expert 740.31
expert 470.26
expert 1270.23
expert 1010.21

The softmax runs over the 4 kept scores only (the model card describes this for gpt-oss): e^(2.3−2.3) = 1, e^(2.14−2.3) = 0.85, e^(2−2.3) = 0.74, e^(1.9−2.3) = 0.67, each divided by their sum. The other 124 experts get weight 0 and are not computed at all.

4 · Run the 4 chosen experts

Expert outputs (each a SwiGLU MLP of x)
d1d2d3d4
expert 741.620.24−0.530.27
expert 47−0.29−0.320.060.15
expert 1270.240.310.24−0.03
expert 1010.100.17−0.040.21

Each expert is a full MLP: down( silu(x·W_gate) * (x·W_up) ). In gpt-oss each one is three 2,880 × 2,880 matrices, 24.9 M parameters, stored in MXFP4 (13.2 MB). gpt-oss also clamps the activation (swiglu_limit: 7.0).

5 · Weighted sum → back into the residual stream

Weights
expert 740.31
expert 470.26
expert 1270.23
expert 1010.21
·
Expert outputs
d1d2d3d4
expert 741.620.24−0.530.27
expert 47−0.29−0.320.060.15
expert 1270.240.310.24−0.03
expert 1010.100.17−0.040.21
=
MoE output
d1d2d3d4
token0.500.10−0.100.16

First number: 0.31·1.62 + 0.26·−0.29 + 0.23·0.24 + 0.21·0.1 = 0.5. This vector is added to the token’s residual stream. Over a 36-layer pass one token touches 4 × 36 = 144 expert-MLPs out of 4,608.

Toy hidden size 4; 128 experts and top-4 as in gpt-oss-120b. Weights are pseudo-random, not learned.

Training keeps the load spread

If the router sent most tokens to a few favourite experts, those would become a bottleneck and the rest would waste memory. Training adds an auxiliary load-balancing loss that penalises uneven use (gpt-oss’s config lists router_aux_loss_coef: 0.9). In inference nothing forces balance; the router does what it learned.

Total versus active parameters

Total Per token (“active”)
Experts 128 × 36 × 24.9 M ≈ 114.7 B 4 × 36 × 24.9 M ≈ 3.6 B
Attention, router, norms ≈ 0.97 B all of it
Embedding + unembedding 1.16 B the unembedding (0.58 B)
Model 116.8 B 5.1 B

Two different costs follow:

  • Compute per token tracks active parameters. About 2 FLOPs per active weight, so roughly 9 GFLOP per token (excluding the unembedding). A dense 117 B model would need about 230 GFLOP.
  • Memory tracks total parameters. All 128 experts must be loaded, because any token may pick any expert. That is why gpt-oss-120b quantizes the experts to MXFP4 (4.25 bits per weight on average): the experts shrink to about 61 GB and the whole checkpoint to 60.8 GiB, so it fits one 80 GB GPU, and leaves an MI355X (288 GB) with over 200 GB for the KV cache.

MXFP4 is a “microscaling” 4-bit floating-point format: each weight is a 4-bit float, and each block of 32 weights shares one 8-bit scale. That is where 4.25 bits per weight comes from (4 + 8/32). MI355X has matrix instructions for MXFP4 and MXFP6 (spec sheet: about 10 PFLOP/s peak). Whether a given server’s kernels run the matrix math in 4-bit or upconvert to 8-bit depends on the kernel library.

What routing does to batching

One expert’s weights are read from memory once per layer per step, then used for every token routed to it in that step. So the useful question is: how many tokens does each expert get?

The walkthrough below simulates it: each token picks 4 of 128 experts at random (the simple model used on this page; real routers are less even). Each grid is one layer’s 128 experts; the number is how many tokens of the step each expert received.

Expert load in one layer, as the batch grows (simulated)

1 · 1 token in the step

Tokens per expert (4 of 128 touched; formula expects 4)
+0+1+2+3+4+5+6+7+8+9+10+11+12+13+14+15
0–150000000000000000
16–310000000000000000
32–470001000000000000
48–631000000101000000
64–790000000000000000
80–950000000000000000
96–1110000000000000000
112–1270000000000000000

One user decoding: 4 experts work, 124 idle. Expert bytes read for the whole 36-layer step: about 1.9 GB.

2 · 8 tokens in the step

Tokens per expert (30 of 128 touched; formula expects 29)
+0+1+2+3+4+5+6+7+8+9+10+11+12+13+14+15
0–151010000100001110
16–310011010101002010
32–470010000000000010
48–630000010001001000
64–791000000020000001
80–950100111000000001
96–1110001100010000000
112–1270010000000000000

8 tokens wake 30 experts, almost all with a single token: 14.3 GB per step, about 8× the single-user read for 8× the tokens. Batching barely shares weight reads yet.

3 · 64 tokens in the step

Tokens per expert (109 of 128 touched; formula expects 111)
+0+1+2+3+4+5+6+7+8+9+10+11+12+13+14+15
0–151550002340141133
16–312032323522352233
32–472102211243020113
48–630213030113112243
64–790132332202422222
80–953303202120532311
96–1111112723114432021
112–1272042401133411131

109 experts touched, each with 2.3 tokens on average (most busy: 7). 51.9 GB per step. A dense model would read the same weights at batch 64 as at batch 1; here the bytes grew about 27×.

4 · 512 tokens in the step

Tokens per expert (128 of 128 touched; formula expects 128)
+0+1+2+3+4+5+6+7+8+9+10+11+12+13+14+15
0–1520172314201221101716191814211816
16–312214121321109143018122014231819
32–471016121718161711161515197111213
48–631016172519122516161616712191317
64–7917151614152118221716161617121713
80–9515171322121619201616242119151815
96–11121161922247141415121271081413
112–1271318141521161713171512920131724

All 128 experts are busy, 16 tokens each on average, most busy 30. The step now reads the whole expert set, 60.9 GB, and further tokens add compute but no bytes. From here on, the bigger the step, the better each byte is used.

5 · Uneven routing: the same 512 tokens, some experts popular

Tokens per expert, skewed routing (busiest: 89; even routing: 30)
+0+1+2+3+4+5+6+7+8+9+10+11+12+13+14+15
0–153529511613151010514172916139
16–311091028139649171387431210
32–4767191117711191132516571514
48–6393505229118104539113518
64–7910123813191814712517511114
80–95912483113631416202103838
96–1112344822331612498953470381020
112–1272622271084562039495220251

Same average (16 per expert), but a few “hot” experts get 89 tokens. On one GPU that matters little. When experts are split across GPUs (below), the GPU holding the hot experts sets the pace for everyone. The skew here is made up to show the effect; real routing statistics depend on the traffic.

Random top-4 routing over 128 experts, seeded so the page is reproducible. Expected experts touched = 128 × (1 − (1 − 4/128)^B). Bytes: touched experts × 36 layers × 13.2 MB (MXFP4).

Try other step sizes:

MoE batch calculator (gpt-oss-120b, random routing)
Experts touched per layer
Tokens per touched expert
Expert bytes read per step
Expert bytes per token
Arithmetic intensity
Ideal step time on one MI355X

Experts touched = 128 × (1 − (1 − 4/128)B). One expert per layer = 3 × 2,880 × 2,880 weights at 4.25 bits = 13.2 MB; 36 layers. Intensity = 2 FLOPs per weight per token ÷ 0.53 bytes per weight × tokens per expert. MI355X: ~5 PFLOP/s dense FP8, ~8 TB/s (spec sheet, approximate). Bar: share of the way to the ridge point (~625 FLOP/byte).

Two lessons:

  1. Decode batching helps MoE less than dense models. Going from 1 to 64 users multiplies the expert bytes read per step by about 27–28× (simulation above, formula in the calculator), because new users wake new experts. Each expert still sees only 2–3 tokens, so arithmetic intensity barely rises. See Decode and Batching and scheduling.
  2. Prefill needs big steps to be compute-bound. An expert does 2 FLOPs per weight per token and costs 0.53 bytes per weight, so its intensity is about 3.8 × tokens per expert. MI355X’s ridge point is about 5 PFLOP/s ÷ 8 TB/s ≈ 625 FLOP/byte (spec sheet, approximate; see roofline). An expert becomes compute-bound at about 165 tokens, which needs a step of about 165 × 128 / 4 ≈ 5,300 tokens.

What it means for short prompts

Short prompts pay a near-fixed cost: every step must read all the experts at least once (about 61 GB at 8 TB/s is 7.6 ms at full bandwidth), plus fixed per-request overhead. Throughput in tokens per second only climbs to its plateau once a step holds several thousand tokens.

Splitting experts across GPUs

When a model does not fit one GPU, or to raise throughput, servers use expert parallelism (EP): different GPUs hold different experts. Each MoE layer then needs two all-to-all exchanges: send each token’s hidden state to the GPUs owning its chosen experts, then gather the results. That traffic runs over the GPU-to-GPU links (XGMI on AMD, NVLink on NVIDIA; see GPU memory and interconnects), and uneven routing makes the busiest GPU set the pace.

gpt-oss-120b fits on one whole MI355X (the ~65 GB checkpoint in 288 GB), so a replica can run without tensor or expert parallelism. Larger MoE models, or NVL72-style racks, are where EP matters.

On the GPU

A naive implementation would loop over experts. Real servers sort tokens by expert and run one grouped GEMM (many small matrix multiplies in one kernel launch), often fused with the routing and the weighted sum (vLLM calls it fused MoE). The kernel view is on Prefill and decode as kernels.

Common misconceptions

  • “Experts are topic specialists (one for code, one for French).” Routing is per token per layer and is learned for loss, not for human topics. Experts do not map cleanly to subjects.
  • “Only 5 B parameters, so it needs 5 B parameters of memory.” It needs all 117 B loaded; only compute shrinks.
  • “MoE makes decode cheap.” Per user, yes: 1.9 GB of expert bytes instead of 61 GB. At a batch of 64, the step reads most of the model anyway.

Further reading