16 hours ago
- The author built a GPT-2 implementation in JAX and found that even without transformer blocks, the token embeddings and output head alone had 77 million parameters due to the large vocabulary size and embedding dimension.
- For the finished 163 million parameter LLM, the input and output components accounted for almost half of the total parameters, which was surprisingly large.
- The feed-forward network (FFN) contains roughly twice as many parameters as the attention layers, yet attention often receives the most focus in explanations of LLM internals.
- In smaller models, token embeddings and the output head dominate the parameter count compared to the transformer layers.
- The author used OpenAI's GPT-5.6 'Sol' variant to create a visualizer that shows parameter breakdowns (embeddings, attention, FFNs, output head) for different GPT-2 sizes, with options for weight tying and QKV bias.
- The visualizer highlights how increasing vocabulary size (e.g., hundreds of thousands of tokens) can make a 'tiny' model consist almost entirely of embeddings and the output head.