- TokenTown visualizes a transformer language model as an isometric city, processing one token at a time.
- The city has districts representing model stages: tokenization, embedding, positional encoding, attention, feed-forward, and sampling.
- The model runs live in the browser with real components like causal masking, KV cache, GELU, and temperature/top-p sampling.
- It is scaled down to 12 dimensions, 2 attention heads, 2–12 layers, and a few hundred vocabulary words.
- Weights are random, so output is blended with a bigram prior and attention is sharpened with biases for readability.
- During a slow tour, each district pauses for 9–26 seconds to show explanations; later, the city runs faster with repeated layers fast-forwarded.
- Controls include Space (pause), S (step), Speed slider (0.4×–8×), and Reset (replay tour) or Run (continue without re-reading).
- The project is inspired by PGSimCity and uses original code, art, and copy.