Most people who use an LLM have a rough sense that "the model reads your text in blocks," but few can say what actually happens between your prompt and the answer. Here is the plain-language version, in numbers you can follow.
Say we have ten words. The tokenizer turns them into twenty tokens:
[154, 27, 891, 42, ..., 517]From here on, the model never thinks in words. It thinks in numbers.
1. Tokens become vectors
The model has an embedding matrix, one row per token in the vocabulary:
Embedding Matrix
[Vocabulary Size × 4096]Each row is a vector for a token:
ID Vector
154 → [0.32, -1.17, ..., 0.11]
27 → [-0.55, 0.81, ..., 0.42]
891 → [ 1.03, -0.27, ..., -0.15]Replace every token with its vector and you get a matrix:
X = [20 × 4096]Twenty tokens, 4096 numbers each. A single matrix. The model now only operates on numbers.
2. The first layer
The model does not process tokens one by one. It multiplies the whole matrix by learned weights:
Q = X × Wq
K = X × Wk
V = X × WvEach gives the same shape:
Q = [20 × 4096]
K = [20 × 4096]
V = [20 × 4096]3. Attention
It then computes:
Q × Kᵀwhich is [20 × 4096] × [4096 × 20] = [20 × 20] — a matrix of relationships between tokens, something like:
cat sits on mat
cat 8 2 1 1
sits 3 7 2 1
on 1 2 6 4
mat 1 1 3 8Each number is how much one token should pay attention to another.
4. Context exchange
Using those weights, the model blends information from the V matrix. Every token picks up context from the others. The token she can become strongly linked to Mary when the context says so.
5. The MLP block
After attention comes another set of transforms:
X
↓
multiply by weight
↓
nonlinearity
↓
another multiplyThis is where the model learns richer features and relationships.
6. Completing the layer
After attention plus the MLP block you get a new matrix — still [20 × 4096] — but with changed contents. The vectors now carry more context.
7. Repeat for every layer
The number of repetitions depends on the number of layers, not on the number of tokens. Twenty tokens form one matrix that passes through every layer:
20 tokens
↓
[20 × 4096]
↓
Layer 1 → Layer 2 → ... → Layer 80An 80-layer model runs the whole sequence through 80 times. The matrix shape stays the same the whole way; only the contents change.
8. Predicting the next token
After the last layer, take the representation of the final token:
h = [4096]Multiply it by the output vocabulary matrix:
logits = h × W_vocab
"cat" 2.1
"dog" 1.5
"house" -0.8
"runs" 8.79. Softmax → pick a token
Turn the raw scores into probabilities:
runs 84%
cat 9%
dog 6%
house 1%The model picks runs, appends it to the sequence, and starts again.
10. The KV cache
To generate the next token, a naive approach would recompute the whole sequence from scratch every time — far too slow.
But during attention every layer computes Q, K, V. For tokens already processed, the K and V values never change. So they can be stored:
Token 1: K1, V1
Token 2: K2, V2
Token 3: K3, V3When a new token arrives, the model computes only its own Q, K, V and compares its Q against the cached K's:
Q4 ↔ K1
Q4 ↔ K2
Q4 ↔ K3
Q4 ↔ K4That stored K/V state is the KV cache (Key-Value Cache). It's why the model does not have to reprocess the entire conversation history for every new token.
The tradeoff is memory: for every token, you hold K and V for every transformer layer. As the context grows, the KV cache becomes a significant chunk of the memory footprint.
The takeaway
Twenty tokens don't become twenty floating objects — they become one [20 × 4096] matrix that flows through layers, attends to itself, exchanges context, and passes an MLP block, dozens of times. Out the other side, the model scores every possible next token and picks the most likely one.
That also explains the cost. The KV cache grows with the prompt, which is exactly why long contexts get expensive — and why keeping a stable prefix (a prompt that reuses the same beginning) lets the model reuse cached K/V and bill far fewer tokens.
