A GPT in slow motion
This is not an animation of a language model — it is one. A tiny, complete GPT runs right here in your browser, and you can watch it think in slow motion: what it sees, what it attends to, what it expects, and how that becomes text.
GPT is short for Generative Pre-trained Transformer: pre-trained on lots of text, generative, built as a transformer — the model architecture behind ChatGPT & co. (more in the glossary).
What you see
A mini-GPT (159,488 parameters, trained only on this site's articles): its context window as a character strip, its attention as arcs, its expectation as probability bars — and the loop that writes.
What it shows
How a language model actually works: read → weigh (attention) → form a distribution → pick ONE character → repeat. The very machinery of the big models, just far smaller.
1 · What it sees — the context window
2 · What it attends to — attention
Attention means just that: the model's way of weighing which parts of the text matter to each other (glossary). The arcs are the last character's eight strongest looks back into the text; the tinting is relative to the strongest spot. Every layer/head pair attends differently — click around.
3 · What it expects — the distribution
Click a bar to take exactly that character — or let the die decide:
4 · And on — the loop
Touch the numbers
Attention of the last character (selected layer/head):
Top 5 with raw logits — the scores before the softmax turns them into probabilities:
Without JavaScript this page can't run the model — but these are real numbers, recorded at export: after the prompt “Der Kontext ” the model expects:
| k | 30.4 % | |
| u | 9.3 % | |
| i | 8.9 % | |
| d | 7.7 % | |
| a | 7.0 % |
What is running here — and what isn't
- A real Transformer (GPT-2 architecture) with exactly 159,488 parameters — the dials that training adjusts (glossary). For scale: GPT-2 (2019) had 1.5 billion; today's frontier models multiples of that.
- Trained exclusively on this website's articles: 26 texts (116,493 characters) for training, two more held back as unseen test material. Whatever wasn't in there does not exist for it — training data decide everything.
- It reads single characters; the big models read word pieces (Tokens) — same mechanics, different grain.
- Expect German-flavoured (and English-flavoured) nonsense: invented words, lost threads. It can even lose its language — to the model, German and English are just character patterns; the big ones stay on track because scale and long context hold them there. At this size that's not a bug — it's pattern machinery in the raw, before scale makes it eloquent.
Under the hood
Architecture: 3 transformer blocks × 4 attention heads, model width 64, context 64 characters, vocabulary 83 characters, tied unembedding. The core of every block:
Attention(Q, K, V) = softmax(Q·KT / √dk) · V
Q, K and V are three learned views of every character — roughly: what am I looking for (Q), what do I offer (K), what do I pass along (V). The softmax turns the match scores into the weights you see as arcs above.
What the big models do differently: word-piece tokens instead of characters, far more blocks, heads and width, refined details (positional encoding, normalisation) — and above all an assistant finish (instruction tuning and RLHF, i.e. refinement from human feedback) plus a training corpus the size of the internet instead of 28 articles.
And why characters here after all? Data volume: the articles yield over 100,000 training examples for 83 characters — as word pieces that would shrink to roughly 40,000 for thousands of distinct pieces, most of them barely ever seen. At this size, characters are the richer statistics — and you get to watch the model build words.
Where to next: why the same machine rolls different answers is in the dice lab; what the big models' reading chunks look like is in the tokens lab; and the vocabulary behind it all is in the glossary.