Learn · LLM inference

Tokens and transformers: what an LLM computes

Updated 10 Nov 2026 · 9 min read

A large language model (LLM) is one function: it takes a list of tokens and returns, for every possible next token, a probability. Generating text means calling that function again and again, appending the chosen token each time. This page walks through one call, using gpt-oss-120b, an open-weight mixture-of-experts model. Its numbers come from its config.json and model card.

The idea

Think of the model as an assembly line that works on a notepad per word.

  • Tokens answer “what are the pieces?” Text is cut into pieces from a fixed list (the vocabulary), and each piece is replaced by its number in that list.
  • Embedding answers “what is a first guess at what this piece means?” Each token number picks one row of a big table: a list of 2,880 numbers, the token’s notepad. Similar pieces get similar notepads.
  • Layers answer “what does this piece mean here?” 36 stations each read every notepad and write a small correction on it. In attention the notepads compare notes with earlier ones (“which earlier words matter to me?”); in the MLP each notepad is processed alone (“given what I now know, what facts apply?”).
  • Unembedding and sampling answer “what comes next?” The last notepad is scored against every vocabulary entry; the scores become probabilities; one token is picked.

Where the analogy breaks: the notepads are not words or facts a person could read. They are vectors of numbers whose meaning is learned, and every layer writes into the same vector (the residual stream) instead of a fresh page.

Vocabulary
201,088
embedding rows (200,019 defined tokens)
Hidden size
2,880
numbers per token vector
Layers
36
18 sliding-window + 18 full attention
Parameters
116.8 B
5.1 B active per token
Context
131,072
tokens at most
Checkpoint
60.8 GiB
experts in MXFP4 (4.25 bits)

Attention: 64 query heads, 8 key/value heads, head size 64. MLP: mixture of experts, 128 experts per layer, 4 used per token.

One forward pass
TRANSFORMER LAYER × 36Text"kubectl get pods"Tokenizero200k_harmony BPE →token idsEmbedding lookupid → vector of 2,880The only step where tokens see each other. Keys and values are kept in the KV cache.RMSNorm →attention64 Q heads, 8 KV heads,RoPEApplied to each token on its own. Holds over 98% of the weights.RMSNorm → MoEMLProuter picks 4 of 128expertsFinal RMSNorm →unembedding2,880 → 201,088 logitsSamplersoftmax, temperature,top-pNext tokentoken idsresidual streamx + attentionx + MLP, after layer 36last position only
  1. TokenizeText is cut into pieces from the 201,088-entry vocabulary. Each piece becomes its integer id.
  2. EmbedEach id selects one row of the embedding table: 2,880 numbers per token. This matrix, tokens × 2,880, is the residual stream.
  3. AttentionRMSNorm, then attention: each token mixes in information from earlier tokens. Keys and values are kept in the KV cache. The result is added to the stream.
  4. MoE MLPRMSNorm, then the router picks 4 of 128 experts per token. Their weighted output is added to the stream. Attention + MLP is one layer; this repeats 36 times.
  5. UnembedAfter layer 36 and a final RMSNorm, the last position’s vector is multiplied by a 2,880 × 201,088 matrix: one score (logit) per vocabulary entry.
  6. SampleSoftmax turns logits into probabilities; the sampler picks one token. Append it and run again: that loop is decode.
gpt-oss-120b shapes. Every token flows through all 36 layers; only the last position's output becomes a next-token choice.

Worked example: one sentence through one layer

The tokenizer below is the real gpt-oss one. The rest is a toy model with vectors of 4 numbers instead of 2,880 and one layer instead of 36; its embeddings and weights are made up, but every number is computed from the ones before it. Step through with the arrows, or switch to Spread to see every step at once.

“The GPU node is ready.” through a toy layer

1 · Tokenize: text → integer ids

"The GPU node is ready." → 6 tokens, 22 characters
  1. The976
  2. ␣GPU47969
  3. ␣node4596
  4. ␣is382
  5. ␣ready6977
  6. .13

The real gpt-oss tokenizer (o200k_harmony, a byte-pair encoding) cuts the sentence into 6 tokens. The leading space belongs to the token: ␣GPU is id 47969, and “The” at the start (976) differs from “␣The” mid-sentence (623). From here on the model sees only these 6 integers.

2 · Embed: each id selects one row

Token ids
The976
␣GPU47969
␣node4596
␣is382
␣ready6977
.13
→ rows of the table →
x: the residual stream (6 tokens × 4)
tokend1d2d3d4
The0.50−0.200.100.80
␣GPU1.200.60−0.400.30
␣node0.900.30−0.800.10
␣is−0.300.700.20−0.50
␣ready0.40−0.601.100.20
.−0.100.200.300.90

Embedding is a lookup, not arithmetic: id 47969 means “take row 47,969 of the table”. In gpt-oss the table is 201,088 × 2,880 (0.58 B numbers) and each row has 2,880 numbers. This tokens × width matrix is the residual stream: every layer reads it and adds to it.

3 · RMSNorm: rescale each row

RMS of each row
The0.48
␣GPU0.72
␣node0.62
␣is0.47
␣ready0.67
.0.49
x ÷ rms =
Normed x (input to attention)
d1d2d3d4
The1.03−0.410.211.65
␣GPU1.680.84−0.560.42
␣node1.450.48−1.290.16
␣is−0.641.500.43−1.07
␣ready0.60−0.901.650.30
.−0.210.410.621.85

Each row is divided by its root-mean-square so it has a typical size of 1 (a learned per-dimension gain follows; it is 1 in this toy). Row “.” : √((−0.1² + 0.2² + 0.3² + 0.9²) ÷ 4) = 0.49. This keeps numbers in range through dozens of layers, and costs a few operations per number.

4 · Attention: each token mixes in earlier tokens

Attention weights (causal)
query ↓ key →The␣GPU␣node␣is␣ready.
The1.00−∞−∞−∞−∞−∞
␣GPU0.840.16−∞−∞−∞−∞
␣node0.650.180.16−∞−∞−∞
␣is0.010.050.020.93−∞−∞
␣ready0.620.230.100.000.05−∞
.0.300.110.020.010.350.21
×
V (values)
d1d2d3d4
The0.52−0.200.110.83
␣GPU0.840.42−0.280.21
␣node0.730.24−0.640.08
␣is−0.320.750.22−0.53
␣ready0.30−0.450.830.15
.−0.100.210.310.93
=
Attention output
d1d2d3d4
The0.52−0.200.110.83
␣GPU0.57−0.100.050.73
␣node0.61−0.02−0.080.59
␣is−0.240.720.18−0.47
␣ready0.60−0.03−0.020.58
.0.34−0.110.350.52

The only step where tokens see each other. Each row of weights says how much of every earlier token’s value to take, and sums to 1. Row “.” takes 0.35 from "␣ready", 0.30 from "The", 0.21 from itself. The upper triangle is masked: no token sees the future. How queries, keys and these weights are computed is on Attention and the KV cache.

5 · Residual add: x + attention

x (embedding)
d1d2d3d4
The0.50−0.200.100.80
␣GPU1.200.60−0.400.30
␣node0.900.30−0.800.10
␣is−0.300.700.20−0.50
␣ready0.40−0.601.100.20
.−0.100.200.300.90
+
Attention output
d1d2d3d4
The0.52−0.200.110.83
␣GPU0.57−0.100.050.73
␣node0.61−0.02−0.080.59
␣is−0.240.720.18−0.47
␣ready0.60−0.03−0.020.58
.0.34−0.110.350.52
=
x after attention
d1d2d3d4
The1.02−0.400.211.63
␣GPU1.770.50−0.351.03
␣node1.510.28−0.880.69
␣is−0.541.420.38−0.97
␣ready1.00−0.631.080.78
.0.240.090.651.42

The block does not replace the stream; it adds a correction. This residual connection is why very deep stacks train at all, and why each layer can be read as “read the stream, write a small update”.

6 · MLP: each token on its own (SwiGLU)

gate = norm(x)·Wgate
d1d2d3d4
The1.48−0.631.011.22
␣GPU2.02−0.440.201.23
␣node1.91−0.83−0.231.36
␣is−0.621.74−0.76−0.84
␣ready0.88−0.631.62−0.05
.1.000.461.130.99
up = norm(x)·Wup
d1d2d3d4
The1.070.68−0.551.67
␣GPU1.011.24−0.561.21
␣node0.701.17−0.961.00
␣is−0.550.311.30−1.24
␣ready1.40−0.20−0.051.33
.0.860.790.281.43
→
silu(gate) × up
d1d2d3d4
The1.29−0.15−0.411.57
␣GPU1.80−0.21−0.061.15
␣node1.16−0.290.101.08
␣is0.120.46−0.310.31
␣ready0.870.04−0.07−0.03
.0.630.220.241.03
× Wdown =
MLP output
d1d2d3d4
The1.08−0.040.130.06
␣GPU0.99−0.220.330.34
␣node0.73−0.120.360.07
␣is0.270.19−0.250.08
␣ready0.27−0.170.040.37
.0.570.190.22−0.04

After another RMSNorm, three matrix multiplies: gate and up, a non-linear silu (z ÷ (1 + e−z)) on the gate as a soft on/off switch, then down. Outlined cell: silu(1) × 0.86 = 0.73 × 0.86 = 0.63. No row looks at another row here. gpt-oss replaces this one MLP with 128 experts and uses 4 per token: Mixture of experts.

7 · Residual add: x + MLP = output of the layer

x after attention
d1d2d3d4
The1.02−0.400.211.63
␣GPU1.770.50−0.351.03
␣node1.510.28−0.880.69
␣is−0.541.420.38−0.97
␣ready1.00−0.631.080.78
.0.240.090.651.42
+
MLP output
d1d2d3d4
The1.08−0.040.130.06
␣GPU0.99−0.220.330.34
␣node0.73−0.120.360.07
␣is0.270.19−0.250.08
␣ready0.27−0.170.040.37
.0.570.190.22−0.04
=
x after the layer
d1d2d3d4
The2.10−0.440.341.69
␣GPU2.760.28−0.021.37
␣node2.240.16−0.520.76
␣is−0.271.610.13−0.89
␣ready1.27−0.801.121.15
.0.810.280.871.38

Same shape in, same shape out: 6 × 4 here, 6 × 2,880 in gpt-oss. That is what lets 36 layers stack. Layer 2 would start again at step 3 with this matrix.

8 · The residual stream of “.”, stage by stage

Row “.” through the layer
d1d2d3d4
embedding x−0.100.200.300.90
+ attention0.34−0.110.350.52
= x after attn0.240.090.651.42
+ MLP0.570.190.22−0.04
= x after layer0.810.280.871.38

One token’s notepad: it starts as its embedding row, gets an attention correction (information from earlier tokens), then an MLP correction. Outlined rows are the stream; the other two are what each block added. After 36 such layers the last row is what predicts the next token.

9 · Predict: last row → logits → probabilities

Final RMSNorm of “.”
d1d2d3d4
.0.880.300.941.50
× Wunembed =
Logits (one per candidate)
␣It␣The␣All␣You↵↵␣Pods
.3.391.912.000.731.39−1.65
softmax →
Probabilities
␣It␣The␣All␣You↵↵␣Pods
.0.5920.1350.1480.0410.0800.004

Only the last position is scored. The unembedding gives one logit per vocabulary entry: 6 candidates here (their ids are real), 201,088 in gpt-oss. Softmax turns them into probabilities; greedy decoding takes “␣It” (id 1225). Earlier positions’ predictions are thrown away in prefill, but their keys and values are kept.

Toy sizes: hidden size 4, one attention head, one dense SwiGLU MLP, 6 candidate output tokens. Each stage is rounded to 2 decimals before the next uses it.

The sampler decides how that distribution becomes one token. Move the sliders:

Sampler: logits → probabilities → one token
token (id)logitlogit ÷ Tprobability
␣It 12253.39
␣The 6231.91
␣All 25452.00
␣You 16080.73
↵↵ 2791.39
␣Pods 113706-1.65

  • Greedy: take the most likely token.
  • Temperature T: divide logits by T before the softmax. Below 1 sharpens, above 1 flattens.
  • Top-p (and top-k): sample only from the smallest set whose probabilities sum to p (or the k most likely).

Appending the chosen token and running again is decode. The first pass over the whole prompt is prefill. A forward pass computes a hidden state for every input position; speculative decoding uses the predictions at several positions at once.

Tokens, in more detail

More real tokenizer output:

"Kubernetes schedules Pods onto nodes." → 7 tokens, 37 characters
  1. K42
  2. ubernetes57796
  3. ␣schedules50216
  4. ␣Pods113706
  5. ␣onto13291
  6. ␣nodes15749
  7. .13
"kubectl get pods -n" → 6 tokens, 19 characters
  1. kub101343
  2. ectl128385
  3. ␣get717
  4. ␣pods88596
  5. ␣-533
  6. n77
"MI355X" → 3 tokens, 6 characters
  1. MI15470
  2. 35531081
  3. X55
  • gpt-oss uses o200k_harmony, a byte-pair encoding (BPE): start from bytes and repeatedly merge the most common adjacent pairs seen in training, so common words become one token and rare words, code and IDs become several.
  • English prose averages roughly 3–4 characters per token. Every cost on these pages (KV cache size, prefill time, price per request) scales with tokens, not characters.
  • Chat requests are wrapped in the harmony format before tokenizing: special tokens mark system, user and assistant turns. A “short” user message still arrives as a few hundred tokens with the system prompt and template. That shared start is what prefix caching reuses.

The layer, in more detail

x = x + Attention( RMSNorm(x) )    # tokens exchange information
x = x + MoE_MLP(   RMSNorm(x) )    # each token is processed on its own
  • RMSNorm: x / sqrt(mean(x²) + 1e-5) * g, applied before each block (“pre-norm”) and once after the last layer.
  • Attention: query, key and value projections per token; scores against earlier keys; softmax; weighted sum of values. 64 query heads of size 64 share 8 key/value heads (grouped-query attention, GQA); an output matrix maps the 4,096 concatenated head outputs back to 2,880.
  • RoPE (rotary position embedding): attention alone does not know token order. RoPE rotates pairs of numbers in each query and key by an angle proportional to the position, so a query–key dot product depends on their distance. gpt-oss uses rope_theta 150,000 and YaRN scaling (factor 32) to stretch a 4,096-token training context to 131,072.
  • Sliding window: every other gpt-oss layer attends only to the last 128 tokens, which halves the long-context KV cache.
  • MLP: SwiGLU, down( silu(gate(x)) * up(x) ), with gate and up mapping 2,880 → 2,880. In gpt-oss each layer has 128 such experts and a router picks 4 per token: 4 × 24.9 M = 99.5 M MLP parameters used per layer, out of 3.2 B. Hence 116.8 B parameters but 5.1 B active. → Mixture of experts
  • Embedding and unembedding: two separate 201,088 × 2,880 matrices (tie_word_embeddings is false).

Where the parameters are

Counting from the config (biases and small tensors left out):

Stored versus used per token (to scale)
Stored: 116.79 B
Used per token: 5.13 B
  • Experts: 114.66 B stored, 3.58 B used
  • Attention: 955.5 M stored, 955.5 M used
  • Router: 13.3 M stored, 13.3 M used
  • Embedding: 579.1 M stored, 2.9 k used
  • Unembedding: 579.1 M stored, 579.1 M used

Computed from the config shapes, without biases and norms. "Used" counts one embedding row and the full unembedding (needed wherever a token is predicted).

Part Shape per layer Parameters (all 36 layers) Used per token
Attention Q, K, V, O 2,880×4,096 + 2×(2,880×512) + 4,096×2,880 ~0.96 B all
Router 2,880 × 128 ~0.01 B all
Experts (SwiGLU) 128 × 3 × 2,880 × 2,880 ~114.7 B 4 of 128 per layer: ~3.6 B
Embedding 201,088 × 2,880 0.58 B one row per token
Unembedding 2,880 × 201,088 0.58 B only where a token is predicted

The model card gives the same split: MLP 114.71 B, attention 0.96 B, embed + unembed 1.16 B, 5.1 B active. Over 98% of the weights are experts, and that is exactly what is stored at 4.25 bits (MXFP4); attention, router and embeddings stay in 16-bit. Hence the 60.8 GiB checkpoint, roughly 65 GB.

For comparison, a dense model such as Llama 3.3 70B (80 layers, hidden size 8,192, 64 query and 8 KV heads of size 128) uses all ~70 B parameters for every token, about 141 GB in 16-bit.

What the hardware actually does

Almost every step is a matrix multiply: a tokens×2,880\text{tokens} \times 2{,}880 activation matrix times a weight matrix. A multiply-add counts as 2 floating-point operations (FLOPs), so a linear layer costs about 2 FLOPs per weight per token.

For gpt-oss-120b, the active weights excluding embeddings are about 4.5 B, so the linear layers cost about 9 GFLOP per token, plus attention, which grows with context length (worked out on Prefill). Llama 3.3 70B costs about 140 GFLOP per token: MoE makes gpt-oss roughly 15× cheaper per token in compute, though it still has to store all 117 B parameters.

The two numbers that decide speed on any chip:

  • FLOPs needed, which matter when many tokens share each weight read (prefill), and
  • bytes read from memory, which matter when few tokens share each weight read (decode).

That split is why prefill and decode behave so differently. The GPU kernels that do these multiplies are on Prefill and decode as kernels.

Common misconceptions

  • “A token is a word.” It is a vocabulary entry; kubectl is two, MI355X three.
  • “Bigger model, slower per token, always.” For MoE models, compute per token tracks active parameters (5.1 B), while memory tracks total (117 B).
  • “The model remembers the conversation.” It does not. Every request re-sends the whole history as tokens. The server can avoid recomputing it with a prefix cache, but that is a cache, not memory in the model.
  • “Only the new token is computed.” In prefill every prompt position runs through every layer. In decode only the new token does, because the KV cache holds the rest.

Further reading