deep learning

Bilingual

Hello Deep Learning: Hyperparameters, inspection, parallelism, ADAM - Bert Hubert's writings
2 months ago
- Parallelization speeds up training by processing batches of inputs simultaneously using GPUs, SIMD, and multi-core CPUs.
- Using optimized matrix multiplication libraries like Eigen can yield dramatic speedups (e.g., 320x).
- Advanced optimizers like ADAM combine momentum and adaptive learning rates to converge much faster and avoid local minima.
- Hyperparameters (learning rate, batch size, momentum) are critical and often require extensive tuning; they differ from learned parameters.
- Inspection tools like SQLiteWriter allow logging and visualization of parameter values and gradients over time.
- ADAM requires storing momentum and velocity tensors for each parameter, and parallelization uses a shared-nothing architecture with thread copies.
Hello Deep Learning - Bert Hubert's writings
2 months ago
- The tutorial provides a GPU-free, from-scratch introduction to modern machine learning.
- It aims to explain concepts deeply without relying on high-level frameworks like PyTorch.
- The tutorial avoids pure mathematical approaches that may only suit seasoned mathematicians.
- The goal is to develop functional neural networks from scratch that yield impressive results on real-life data.
- The tutorial includes code and Markdown files hosted on GitHub for collaborative improvements.
- It covers 12 chapters, starting from linear combinations to advanced topics like attention and transformers, including practical applications like handwritten digit recognition and language processing.
Hello Deep Learning: Intro - Bert Hubert's writings
2 months ago
- Author previously dismissed deep learning as overhyped but later realized significant advances were happening.
- Inspired by Andrej Karpathy's 2015 post on recurrent neural networks, author decided to catch up.
- Author aims to build a self-contained, from-scratch demo in plain C++ to explain deep learning fundamentals.
- The project will cover techniques like recognizing handwritten letters using real data.
- The code will use only system libraries plus logging, matrix, and image processing libraries.
- Goal is to show what deep learning tooling does under the hood, not just how to use it.
Hello Deep Learning: actually learning something - Bert Hubert's writings
2 months ago
- The chapter demonstrates how a neural network can learn to distinguish between images of digits 3 and 7 by itself, using a simple weight adjustment technique.
- Initially, weights are randomly initialized and produce incorrect scores; the learning process adjusts weights in the direction of the correct output.
- The adjustment is proportional to the input pixel values (bright pixels matter more) and controlled by a learning rate (e.g., 0.01).
- This simple approach, called backpropagation, achieves over 98% accuracy on clean test data, comparable to the earlier hand-configured network.
- The code initializes a random weights matrix, applies a learning rate via an identity matrix, and updates weights and bias after each training sample based on the margin of error.
- The process avoids infinite weight growth by using thresholds (2 or -2) for updates, which will later be replaced by activation functions.
Hello Deep Learning: Automatic differentiation, autograd - Bert Hubert's writings
2 months ago
- The chapter introduces a 5-layer neural network capable of recognizing all 10 handwritten digits with near-perfect accuracy, using 109184 weights and 202 biases.
- Learning involves gradient descent, where derivatives of the error with respect to all parameters are computed and used to adjust parameters by a small learning rate.
- Automatic differentiation (autograd) is a core technique that efficiently computes derivatives by building a directed acyclic graph (DAG) from operations, then traversing it topologically to propagate gradients.
- The implementation relies on operator overloading and reference counting (e.g., in C++) to lazily construct the DAG, enabling forward and backward passes for training.
- Key components include the Tensor class with Eigen for fast matrix operations, and methods like 'assureValue' for forward pass and 'doGrad' for backpropagation.
Hello Deep Learning: Convolutional networks - Bert Hubert's writings
2 months ago
- Convolutional networks (CNNs) use kernels to scan for features in images, making them robust to pixel position changes.
- Max-pooling reduces sensitivity to feature location by downsampling, using non-overlapping windows to keep only the maximum values.
- GELU activation function often outperforms ReLU by retaining more information from negative inputs.
- The described CNN design for handwritten letter recognition includes three convolutional+max-pooling stages, followed by three fully connected layers and LogSoftMax.
- Training convolutional networks is computationally intensive (e.g., can take a day) due to many operations, though parallel processing helps.
- Overfitting is a key challenge; production use typically requires progressing through training, validation, real-world data, and reproducibility.
- The network's performance on handwritten letters can be subjective (e.g., g vs. q), leading to about 85% accuracy but over 95% when considering second-best guesses.
- Backpropagation in CNNs involves sliding kernels over inputs and gradients to update weights and biases.
- Actual implementation swaps digit datasets for letter datasets and uses a ConvoAlphabetModel instead of a simpler digit model.
Hello Deep Learning: Dropout, data augmentation, weight decay and quantisation - Bert Hubert's writings
2 months ago
- Data augmentation exposes the network to more variations by modifying inputs, improving robustness and compensating for limited training data.
- Normalization standardizes inputs (e.g., mean and standard deviation) to ensure consistency with training conditions.
- Dropout randomly zeroes out parts of the network during training, forcing multiple pathways and preventing overfitting.
- Weight decay adds squared parameter values to the loss function, driving unused parameters toward zero and simplifying the network.
- Quantization reduces parameter precision (e.g., to 16 or 4 bits) to save memory and power, often with minimal performance loss.
Hello Deep Learning: Further reading & worthwhile projects - Bert Hubert's writings
2 months ago
- Completing the blog series provides a solid understanding of deep learning basics, but there's much more to learn beyond this 20k-word introduction.
- Recommended resources include PyTorch documentation, Andrej Karpathy's micrograd and RNN posts, and FastAI's Jupyter notebooks.
- Notable projects for further learning: Whisper.cpp (speech recognition in C++) and Llama.cpp (large language model), both runnable on modest hardware.
- The author invites readers to contribute recommendations via pull requests or email.
November newsletter
2 months ago
- Links on deep learning and genomics scaling
- Dark mode rewrite of the site
- One essay published
- One opera review of Wagner's 'The Ring' cycle
Show HN: FeyNoBg – Automatic background removal model and training library
2 months ago
- FeyNoBg is a state-of-the-art model for automatic background removal, achieving the best S-measure on four of eight benchmarks and within 2% on the remaining four.
- The model requires two complementary skills: foreground recognition (separating subject from background) and boundary precision (image matting for edges like hair or motion blur).
- FeyNoBg builds on BiRefNet architecture, expanding the third feature extraction stage from 18 to 24 blocks (263M parameters) to improve information retention without forgetting prior knowledge.
- A diverse training dataset of 26.1K images from 10 sources was assembled, with each source capped at 4,000 images and annotations converted to binary masks to ensure consistency.
- The NoBg library provides a unified Python interface for running and training background removal models, offering higher throughput and lower latency than the original BiRefNet implementation.
- Both FeyNoBg and NoBg are open source, available on Hugging Face and GitHub respectively.
SEAM-V: A Hybrid-Decoupled RISC-V Vector Processor
2 months ago
- The paper introduces SEAM-V, a hybrid-decoupled RISC-V vector execution architecture designed to address limitations in tightly coupled vector implementations.
- SEAM-V uses task-level decoupling, local instruction supply, and VLIW-style packing to form continuous streams of execute packets (EPs), improving instruction supply and reducing scalar-side delays.
- EP identity and request-bound prefetch context remain visible to the dynamic vector backend, enabling same-EP hazard suppression and request-bound prefetching to enhance sustained throughput.
- The architecture supports limited cross-EP vector overlap under safety conditions, while cross-EP dependences, resource conflicts, and memory ordering are dynamically managed by the backend.
- Cycle-accurate RTL evaluation shows SEAM-V achieves a geometric-mean speedup of 1.34x over an Ara-based tightly coupled implementation across 17 kernels, with speedups of 1.50x, 1.25x, and 1.27x for variable-AVL, BLAS/matrix, and fixed-size application groups, respectively.
- At AVL=32, the geometric-mean speedup across six one-dimensional vector kernels approaches 3x.
Wax On, Weights Off: Knowledge Distillation Explained
2 months ago
- Knowledge distillation is a technique where a small student model learns from a large teacher model's outputs, compressing years of R&D into weeks.
- Three types exist: response-based (matching outputs), feature-based (matching internal representations), and relation-based (matching relationships between examples); only response-based is feasible via API for closed models.
- Distillation against a closed model is a distributed attack using thousands of API accounts, query diversification (self-instruct loops, Evol-Instruct), and data engineering to harvest and clean training data.
- The pipeline is primarily a data engineering system: prompt generation, collection (with evasion of defenses), curation (filtering/quality scoring), and training (SFT, LoRA, preference optimization).
- Economic advantage: distillation pays only for compression, skipping costly discovery (data curation, architecture search, alignment), enabling fast-follower catch-up at lower cost.
- Frontier developer tech stack includes pretraining on massive data, alignment (RLHF, reward modeling), evaluation, and inference defense—none of which a distiller invests in unless selling a model.
- Distillation commoditizes the teacher's moat but forces the student into a perpetual catch-up position, inheriting the teacher's biases and capability ceiling.
- The US-China AI race is reframed by distillation: Chinese labs like DeepSeek, Moonshot, and MiniMax have been caught industrial-scale harvesting of Claude, leading to access restrictions and geopolitical tensions.
From Muon to Gradient Clipping: Some Thoughts on QK Stability
2 months ago
- Muon optimizer works in function space by constraining the spectral norm of weight updates, aiming to limit the functional change for any input, contrasting with parameter-space optimizers like Adam.
- Applying standard Muon to Transformer's Q and K matrices leads to training instability because it separately constrains updates to W_Q and W_K, failing to control their coupled product in attention scores, which can cause spectral norm explosion.
- A principled attempt to modify Muon for QK involves directly constraining the change in attention score matrix S, leading to an optimization problem with a constraint coupling ΔW_Q and ΔW_K.
- Decoupling this constraint via the triangle inequality and solving subproblems results in a theoretically clean update formula, but it requires expensive pseudoinverse computations, making it impractical.
- A mathematical simplification reveals that the gradient G_Q naturally contains W_K, leading to a geometric interpretation involving a projection matrix, yet the external pseudoinverse remains, not solving the computational bottleneck.
- Viewing attention through a unified bilinear matrix B = W_Q W_K^T simplifies the Muon update to a single-matrix constraint, but this approach is impractical due to parameter explosion (d^2 vs. 2dh parameters).
- Multi-head attention can be seen as a low-rank approximation of full bilinear attention, connecting to LoRA's structure, suggesting potential for a Muon-LoRA optimizer for parameter-efficient fine-tuning.
- Due to computational hurdles, practical solutions shift to empirical approximations like gradient clipping or Kimi's MuonClip, which use batch information to approximate worst-case behavior and stabilize training.
Theories of Deep Learning
2 months ago
- Deep learning has seen exponential empirical success, but theory lagged behind; however, the gap is narrowing with multiple emerging theories.
- Theories in deep learning are categorized into architecture theory, optimization theory, and functional theory, each focusing on different aspects like model design, optimizer behavior, and generalization.
- Categorical Deep Learning uses category theory to generalize all neural network architectures, providing a bridge between constraints and implementations via monads in a 2-category.
- Modular Duality offers an optimization framework based on norms to improve training efficiency, introducing duality maps to align gradient updates with parameter spaces.
- A Theory of Generalization shifts focus from parameter space to output space, using the empirical neural tangent kernel to explain phenomena like benign overfitting, double descent, implicit bias, and grokking.
- Other research areas include mechanistic interpretability and various explanatory principles, but the highlighted theories provide strong foundational frameworks.
FlashAttention-4: Algorithm and Kernel Pipelining
3 months ago
- Asymmetric hardware scaling on modern GPUs like Blackwell leads to tensor core throughput growing faster than resources like shared memory bandwidth and special function units (SFUs).
- FlashAttention-4 is a co-designed algorithm and kernel for the Blackwell architecture, achieving up to 1605 TFLOPs/s on B200 with BF16, outperforming cuDNN and Triton.
- Key innovations include new pipelining for overlap, software emulation of exponential via polynomial approximation to mitigate SFU bottlenecks, and TMEM usage to reduce shared memory traffic.
- Blackwell hardware features like Tensor Memory (TMEM), fully asynchronous 5th gen tensor cores, and 2-CTA MMA mode enable larger tiles, reduced traffic, and improved performance.
- The backward pass is shared-memory bandwidth limited; optimizations include 2-CTA MMA to halve traffic and reduce atomic adds, plus deterministic mode for reproducible training.
- Scheduling improvements address load imbalance from causal masking and variable sequence lengths via grid linearization and longest-processing-time-first (LPT) scheduling.
- FlashAttention-4 is implemented in CuTe-DSL, cutting compile times significantly, and has influenced optimizations in cuDNN versions 9.13 and later.
Scaling Laws, Carefully
3 months ago
- Scaling laws describe power-law relationships between training loss and model size (N), dataset size (D), and compute (C), essential for optimal compute allocation in deep learning.
- Early works (e.g., Amari et al. 1992, Hestness et al. 2017) established power-law learning curves and that architecture affects offset but not exponent, which is domain-specific.
- Kaplan et al. (2020) formalized scaling laws for Transformers, suggesting model size should grow faster than data (N_opt ∝ C^0.73), but this was later contested.
- Chinchilla (Hoffmann et al. 2022) found optimal scaling with N_opt ∝ C^0.5, advocating for smaller models trained on more tokens, showing many large models were undertrained.
- Discrepancies between Kaplan and Chinchilla arise from differences in scale, embedding parameter accounting, and fitting methods, later reconciled by Pearce & Song (2024).
- Power laws may stem from data manifold dimensionality or quantized skill learning, but theoretical explanations remain incomplete.
- In data-limited regimes, studies (e.g., Hernandez et al. 2022, Muennighoff et al. 2023) show repeated data harms efficiency, with models adjusted via effective data and overfitting penalties.
- Lovelace et al. (2026) introduced an overfitting penalty term based on capacity ratio (N/U_D) and repetition, improving scaling law fits for constrained data.
- Fitting scaling laws is tricky due to sensitivity to loss precision, noise, fitting region, and assumptions like fixed architecture and tuning, as seen in replication attempts (Besiroglu et al. 2024).

deep learning

Hello Deep Learning: Hyperparameters, inspection, parallelism, ADAM - Bert Hubert's writings

Hello Deep Learning - Bert Hubert's writings

Hello Deep Learning: Intro - Bert Hubert's writings

Hello Deep Learning: actually learning something - Bert Hubert's writings

Hello Deep Learning: Automatic differentiation, autograd - Bert Hubert's writings

Hello Deep Learning: Convolutional networks - Bert Hubert's writings

Hello Deep Learning: Dropout, data augmentation, weight decay and quantisation - Bert Hubert's writings

Hello Deep Learning: Further reading & worthwhile projects - Bert Hubert's writings

November newsletter

Show HN: FeyNoBg – Automatic background removal model and training library

SEAM-V: A Hybrid-Decoupled RISC-V Vector Processor

Wax On, Weights Off: Knowledge Distillation Explained

From Muon to Gradient Clipping: Some Thoughts on QK Stability

Theories of Deep Learning

FlashAttention-4: Algorithm and Kernel Pipelining

Scaling Laws, Carefully