A GPT in slow motion
This is not an animation of a language model — it is one. A tiny, complete GPT runs right here in your browser, and you can watch it think in slow motion: what it sees, what it attends to, what it expects, and how that becomes text.
GPT is short for Generative Pre-trained Transformer: pre-trained on lots of text, generative, built as a transformer — the model architecture behind ChatGPT & co. (more in the glossary).
What you see
A mini-GPT (159,488 parameters, trained only on this site's articles): its context window as a character strip, its attention as arcs, its expectation as probability bars — and the loop that writes.
What it shows
How a language model actually works: read → weigh (attention) → form a distribution → pick ONE character → repeat. The very machinery of the big models, just far smaller.
1 · What it sees — the context window
2 · What it attends to — attention
Attention means just that: the model's way of weighing which parts of the text matter to each other (glossary). The arcs are the last character's eight strongest looks back into the text; the tinting is relative to the strongest spot. Every layer/head pair attends differently — click around.
3 · What it expects — the distribution
Click a bar to take exactly that character — or let the die decide:
4 · And on — the loop
Touch the numbers
Attention of the last character (selected layer/head):
Top 5 with raw logits — the scores before the softmax turns them into probabilities:
Without JavaScript this page can't run the model — but these are real numbers, recorded at export: after the prompt “Der Kontext ” the model expects:
| k | 17.0 % | |
| d | 10.6 % | |
| – | 7.5 % | |
| a | 5.8 % | |
| e | 5.8 % |
What is running here — labelled honestly
- A real Transformer (GPT-2 architecture) with exactly 159,488 parameters — the dials that training adjusts (glossary). For scale: GPT-2 (2019) had 1.5 billion; today's frontier models multiples of that.
- Trained exclusively on the 28 article texts of this website (135,298 characters). Whatever wasn't in there does not exist for it — training data decide everything.
- It reads single characters; the big models read word pieces (Tokens) — same mechanics, different grain.
- Expect German-flavoured (and English-flavoured) nonsense: invented words, lost threads. At this size that's not a bug — it's an honest look at pure pattern machinery, before scale makes it eloquent.
Under the hood
Architecture: 3 transformer blocks × 4 attention heads, model width 64, context 64 characters, vocabulary 83 characters, tied unembedding. The core of every block:
Attention(Q, K, V) = softmax(Q·KT / √dk) · V
Q, K and V are three learned views of every character — roughly: what am I looking for (Q), what do I offer (K), what do I pass along (V). The softmax turns the match scores into the weights you see as arcs above.
What the big models do differently: word-piece tokens instead of characters, far more blocks, heads and width, refined details (positional encoding, normalisation) — and above all an assistant finish (instruction tuning and RLHF, i.e. refinement from human feedback) plus a training corpus the size of the internet instead of 28 articles.
Where to next: why the same machine rolls different answers is in the dice lab; what the big models' reading chunks look like is in the tokens lab; and the vocabulary behind it all is in the glossary.