Show HN: Tiny Diffusion – A character-level text diffusion model from scratch
12 days ago
- #text-generation
- #language-model
- #diffusion
- A character-level language diffusion model for text generation, based on nanochat GPT and trained on Tiny Shakespeare.
- Model size: 10.7 million parameters, 6 layers, 6 attention heads, 384 embedding dim, 256 sequence length, 128 diffusion steps.
- Repository includes training, sampling, and animation scripts.
- Pre-trained weights are provided (weights/diffusion_model.pt).
- Training took ~30 minutes on 4xA100s for 20,000 steps.
- Sample generation and diffusion process visualization available.
- Includes a Game of Life-inspired sampling experiment.
- Directory structure: model.py, training.py, sample.py, data/, weights/, animations/.