Wax On, Weights Off: Knowledge Distillation Explained

13 hours ago

Knowledge distillation is a technique where a small student model learns from a large teacher model's outputs, compressing years of R&D into weeks.
Three types exist: response-based (matching outputs), feature-based (matching internal representations), and relation-based (matching relationships between examples); only response-based is feasible via API for closed models.
Distillation against a closed model is a distributed attack using thousands of API accounts, query diversification (self-instruct loops, Evol-Instruct), and data engineering to harvest and clean training data.
The pipeline is primarily a data engineering system: prompt generation, collection (with evasion of defenses), curation (filtering/quality scoring), and training (SFT, LoRA, preference optimization).
Economic advantage: distillation pays only for compression, skipping costly discovery (data curation, architecture search, alignment), enabling fast-follower catch-up at lower cost.
Frontier developer tech stack includes pretraining on massive data, alignment (RLHF, reward modeling), evaluation, and inference defense—none of which a distiller invests in unless selling a model.
Distillation commoditizes the teacher's moat but forces the student into a perpetual catch-up position, inheriting the teacher's biases and capability ceiling.
The US-China AI race is reframed by distillation: Chinese labs like DeepSeek, Moonshot, and MiniMax have been caught industrial-scale harvesting of Claude, leading to access restrictions and geopolitical tensions.

Hasty Briefsbeta

#knowledge distillation

Knowledge Distillation of Black-Box Large Language Models (2024)

Wax On, Weights Off: Knowledge Distillation Explained