Bilingual

Scaling Laws, Carefully
a month ago
- Scaling laws describe power-law relationships between training loss and model size (N), dataset size (D), and compute (C), essential for optimal compute allocation in deep learning.
- Early works (e.g., Amari et al. 1992, Hestness et al. 2017) established power-law learning curves and that architecture affects offset but not exponent, which is domain-specific.
- Kaplan et al. (2020) formalized scaling laws for Transformers, suggesting model size should grow faster than data (N_opt ∝ C^0.73), but this was later contested.
- Chinchilla (Hoffmann et al. 2022) found optimal scaling with N_opt ∝ C^0.5, advocating for smaller models trained on more tokens, showing many large models were undertrained.
- Discrepancies between Kaplan and Chinchilla arise from differences in scale, embedding parameter accounting, and fitting methods, later reconciled by Pearce & Song (2024).
- Power laws may stem from data manifold dimensionality or quantized skill learning, but theoretical explanations remain incomplete.
- In data-limited regimes, studies (e.g., Hernandez et al. 2022, Muennighoff et al. 2023) show repeated data harms efficiency, with models adjusted via effective data and overfitting penalties.
- Lovelace et al. (2026) introduced an overfitting penalty term based on capacity ratio (N/U_D) and repetition, improving scaling law fits for constrained data.
- Fitting scaling laws is tricky due to sensitivity to loss precision, noise, fitting region, and assumptions like fixed architecture and tuning, as seen in replication attempts (Besiroglu et al. 2024).
FlashAttention-4: Algorithm and Kernel Pipelining
22 days ago
- Asymmetric hardware scaling on modern GPUs like Blackwell leads to tensor core throughput growing faster than resources like shared memory bandwidth and special function units (SFUs).
- FlashAttention-4 is a co-designed algorithm and kernel for the Blackwell architecture, achieving up to 1605 TFLOPs/s on B200 with BF16, outperforming cuDNN and Triton.
- Key innovations include new pipelining for overlap, software emulation of exponential via polynomial approximation to mitigate SFU bottlenecks, and TMEM usage to reduce shared memory traffic.
- Blackwell hardware features like Tensor Memory (TMEM), fully asynchronous 5th gen tensor cores, and 2-CTA MMA mode enable larger tiles, reduced traffic, and improved performance.
- The backward pass is shared-memory bandwidth limited; optimizations include 2-CTA MMA to halve traffic and reduce atomic adds, plus deterministic mode for reproducible training.
- Scheduling improvements address load imbalance from causal masking and variable sequence lengths via grid linearization and longest-processing-time-first (LPT) scheduling.
- FlashAttention-4 is implemented in CuTe-DSL, cutting compile times significantly, and has influenced optimizations in cuDNN versions 9.13 and later.
Theories of Deep Learning
14 days ago
- Deep learning has seen exponential empirical success, but theory lagged behind; however, the gap is narrowing with multiple emerging theories.
- Theories in deep learning are categorized into architecture theory, optimization theory, and functional theory, each focusing on different aspects like model design, optimizer behavior, and generalization.
- Categorical Deep Learning uses category theory to generalize all neural network architectures, providing a bridge between constraints and implementations via monads in a 2-category.
- Modular Duality offers an optimization framework based on norms to improve training efficiency, introducing duality maps to align gradient updates with parameter spaces.
- A Theory of Generalization shifts focus from parameter space to output space, using the empirical neural tangent kernel to explain phenomena like benign overfitting, double descent, implicit bias, and grokking.
- Other research areas include mechanistic interpretability and various explanatory principles, but the highlighted theories provide strong foundational frameworks.
From Muon to Gradient Clipping: Some Thoughts on QK Stability
12 days ago
- Muon optimizer works in function space by constraining the spectral norm of weight updates, aiming to limit the functional change for any input, contrasting with parameter-space optimizers like Adam.
- Applying standard Muon to Transformer's Q and K matrices leads to training instability because it separately constrains updates to W_Q and W_K, failing to control their coupled product in attention scores, which can cause spectral norm explosion.
- A principled attempt to modify Muon for QK involves directly constraining the change in attention score matrix S, leading to an optimization problem with a constraint coupling ΔW_Q and ΔW_K.
- Decoupling this constraint via the triangle inequality and solving subproblems results in a theoretically clean update formula, but it requires expensive pseudoinverse computations, making it impractical.
- A mathematical simplification reveals that the gradient G_Q naturally contains W_K, leading to a geometric interpretation involving a projection matrix, yet the external pseudoinverse remains, not solving the computational bottleneck.
- Viewing attention through a unified bilinear matrix B = W_Q W_K^T simplifies the Muon update to a single-matrix constraint, but this approach is impractical due to parameter explosion (d^2 vs. 2dh parameters).
- Multi-head attention can be seen as a low-rank approximation of full bilinear attention, connecting to LoRA's structure, suggesting potential for a Muon-LoRA optimizer for parameter-efficient fine-tuning.
- Due to computational hurdles, practical solutions shift to empirical approximations like gradient clipping or Kimi's MuonClip, which use batch information to approximate worst-case behavior and stabilize training.
Wax On, Weights Off: Knowledge Distillation Explained
5 days ago
- Knowledge distillation is a technique where a small student model learns from a large teacher model's outputs, compressing years of R&D into weeks.
- Three types exist: response-based (matching outputs), feature-based (matching internal representations), and relation-based (matching relationships between examples); only response-based is feasible via API for closed models.
- Distillation against a closed model is a distributed attack using thousands of API accounts, query diversification (self-instruct loops, Evol-Instruct), and data engineering to harvest and clean training data.
- The pipeline is primarily a data engineering system: prompt generation, collection (with evasion of defenses), curation (filtering/quality scoring), and training (SFT, LoRA, preference optimization).
- Economic advantage: distillation pays only for compression, skipping costly discovery (data curation, architecture search, alignment), enabling fast-follower catch-up at lower cost.
- Frontier developer tech stack includes pretraining on massive data, alignment (RLHF, reward modeling), evaluation, and inference defense—none of which a distiller invests in unless selling a model.
- Distillation commoditizes the teacher's moat but forces the student into a perpetual catch-up position, inheriting the teacher's biases and capability ceiling.
- The US-China AI race is reframed by distillation: Chinese labs like DeepSeek, Moonshot, and MiniMax have been caught industrial-scale harvesting of Claude, leading to access restrictions and geopolitical tensions.
SEAM-V: A Hybrid-Decoupled RISC-V Vector Processor
4 days ago
- The paper introduces SEAM-V, a hybrid-decoupled RISC-V vector execution architecture designed to address limitations in tightly coupled vector implementations.
- SEAM-V uses task-level decoupling, local instruction supply, and VLIW-style packing to form continuous streams of execute packets (EPs), improving instruction supply and reducing scalar-side delays.
- EP identity and request-bound prefetch context remain visible to the dynamic vector backend, enabling same-EP hazard suppression and request-bound prefetching to enhance sustained throughput.
- The architecture supports limited cross-EP vector overlap under safety conditions, while cross-EP dependences, resource conflicts, and memory ordering are dynamically managed by the backend.
- Cycle-accurate RTL evaluation shows SEAM-V achieves a geometric-mean speedup of 1.34x over an Ara-based tightly coupled implementation across 17 kernels, with speedups of 1.50x, 1.25x, and 1.27x for variable-AVL, BLAS/matrix, and fixed-size application groups, respectively.
- At AVL=32, the geometric-mean speedup across six one-dimensional vector kernels approaches 3x.

deep learning

Scaling Laws, Carefully

FlashAttention-4: Algorithm and Kernel Pipelining

Theories of Deep Learning

From Muon to Gradient Clipping: Some Thoughts on QK Stability

Wax On, Weights Off: Knowledge Distillation Explained

SEAM-V: A Hybrid-Decoupled RISC-V Vector Processor