Bilingual

Hello Deep Learning - Bert Hubert's writings
a day ago
- The tutorial provides a GPU-free, from-scratch introduction to modern machine learning.
- It aims to explain concepts deeply without relying on high-level frameworks like PyTorch.
- The tutorial avoids pure mathematical approaches that may only suit seasoned mathematicians.
- The goal is to develop functional neural networks from scratch that yield impressive results on real-life data.
- The tutorial includes code and Markdown files hosted on GitHub for collaborative improvements.
- It covers 12 chapters, starting from linear combinations to advanced topics like attention and transformers, including practical applications like handwritten digit recognition and language processing.
Hello Deep Learning: Linear combinations - Bert Hubert's writings
a day ago
- - The 'Hello Deep Learning' series introduces neural networks using the EMNIST dataset of handwritten digits (3 and 7).
- - A simple neural network layer (linear combination) is manually configured using the difference between average images of 3s and 7s as weights.
- - By setting a decision rule based on a bias-adjusted score, the model achieves 97.025% accuracy on validation data.
- - Processing includes averaging tensors, computing Hadamard products, and using matrix multiplication for scoring.
- - The code implementation (threeorseven.cc) preprocesses data with MNISTReader, computes weights, applies bias, and evaluates accuracy.
Hello Deep Learning: Automatic differentiation, autograd - Bert Hubert's writings
a day ago
- The chapter introduces a 5-layer neural network capable of recognizing all 10 handwritten digits with near-perfect accuracy, using 109184 weights and 202 biases.
- Learning involves gradient descent, where derivatives of the error with respect to all parameters are computed and used to adjust parameters by a small learning rate.
- Automatic differentiation (autograd) is a core technique that efficiently computes derivatives by building a directed acyclic graph (DAG) from operations, then traversing it topologically to propagate gradients.
- The implementation relies on operator overloading and reference counting (e.g., in C++) to lazily construct the DAG, enabling forward and backward passes for training.
- Key components include the Tensor class with Eigen for fast matrix operations, and methods like 'assureValue' for forward pass and 'doGrad' for backpropagation.
Hello Deep Learning: Reading handwritten digits - Bert Hubert's writings
a day ago
- The network architecture flattens a 28x28 image to 784x1, then applies three linear layers (128, 64, and 10 units) with ReLU activations between the first two layers.
- LogSoftMax is applied to the final 10 outputs to convert them into log probabilities, and cross-entropy loss is computed by comparing the predicted log probability for the correct digit (via a one-hot vector) to zero.
- Training uses stochastic gradient descent (SGD) with batches of 64 images, accumulating gradients over the batch and updating parameters with a learning rate divided by batch size.
- The model achieves 92.23% accuracy on the test set, with a confusion matrix showing common confusions (e.g., between 7 and 9) but no confusion between digits like 7 and 6.
- The network is heavily dependent on pixel positions and does not capture shape relations, limiting its generalization to uncentered or noisy digits.
Hello Deep Learning: Dropout, data augmentation, weight decay and quantisation - Bert Hubert's writings
a day ago
- Data augmentation exposes the network to more variations by modifying inputs, improving robustness and compensating for limited training data.
- Normalization standardizes inputs (e.g., mean and standard deviation) to ensure consistency with training conditions.
- Dropout randomly zeroes out parts of the network during training, forcing multiple pathways and preventing overfitting.
- Weight decay adds squared parameter values to the loss function, driving unused parameters toward zero and simplifying the network.
- Quantization reduces parameter precision (e.g., to 16 or 4 bits) to save memory and power, often with minimal performance loss.
We Need a Neural Network (2006)
7 days ago
- M.A., a neural network expert, recommended against using neural networks for a data parsing task, but management insisted.
- He spent a year developing a complex, inefficient neural network system that required a server cluster and extensive training.
- After three years, new management replaced the system with a simpler traditional solution that ran on a single server.
- The old neural network was repurposed to generate prose for a newsletter, producing a strange, poetic output.
Overtraining as the path to human-like AI
10 days ago
- Gwern proposes that overtraining large models on small datasets could lead to grokking, enabling human-like generalization in AI.
- Grokking is a phenomenon where prolonged training forces models to find simpler, deeper understandings of data, beyond mere memorization.
- Current AI labs focus on training smaller models on vast datasets, which may hinder grokking and limit generalization capabilities.
- Gwern suggests training a massive model (e.g., 100 trillion parameters) on a constrained dataset to encourage deeper learning, though this approach is untested and risky.
- The idea faces technical and political challenges, including high costs and the appearance of failure during training until a breakthrough occurs.
Latent Space as a New Medium
15 days ago
- The latent space within AI models serves as a new creative medium, compressing human knowledge into a high-dimensional map.
- LLMs achieve extreme compression by abstracting common attributes of all data, storing patterns without storing the original data.
- Latent space integrates everything—real and possible—into a single continuous map, enabling creative exploration across domains.
- Answers from LLMs are generated through journeys in latent space, where correctness and truth are directional vectors.
- Latent space allows for easy manipulation of ideas and styles, as seen in image generators transferring artistic styles.
- Potential applications include prototyping with attributes, discovering white spaces between known concepts, and cross-domain analogies.
- It enables new forms of measurement, anomaly detection, simulated realities, and personal AI models for co-creation.
- Latent spaces could evolve into fields studying their architectures, with future uses in world-building and innovation beyond current imagination.
A Theory of Contrastive Learning with Natural Images
20 days ago
- The paper proposes a theory explaining why contrastive learning with simple images and augmentations yields useful representations for downstream tasks.
- It analytically computes the optimal representation for a contrastive loss given basic augmentations and any image dataset with stationary statistics.
- For certain augmentations, the optimum can be achieved by a CNN with sinusoidal first-layer filters, a pointwise nonlinearity, global average pooling, and a final linear layer performing partial whitening.
- Even with more complex augmentations, the optimal weights in such CNNs remain sinusoidal, with frequencies and weights computable via a waterfilling algorithm based on the dataset's expected power spectrum.
- Experiments with various image datasets and augmentations confirm that CNNs trained with SGD empirically learn sinusoidal filters and partial whitening, aligning with the theoretical predictions.
Neural Geometry in Vision Models with Block-Sparse Featurizers
21 days ago
- Block-Sparse Featurizers (BSF) decompose model activations into multidimensional subspaces instead of single directions, capturing concepts as manifolds.
- BSFs outperform standard sparse autoencoders (SAEs) in recovering interpretable features and explaining model activations more efficiently, as shown in toy models and vision models like DINOv3.
- Features found by BSFs enable fine-grained steering, allowing exploration of concept variations, such as different types of trees or pretzels, by moving within the multidimensional subspaces.
- Most concepts in vision models are multidimensional, typically two- to four-dimensional, challenging the assumption that concepts are represented as one-dimensional lines.
- BSFs reveal continuous structures in neural networks, such as curve detectors in InceptionV1, uncovering symmetries like Fourier harmonics that were previously fragmented.
- The principle of block-sparsity is key, emphasizing that subspaces fire together, and different BSF variants (vanilla, Grassmannian, group-lasso) implement this with varying encoder/decoder designs.
- Future work may involve adapting featurizers to different domains, as block-sparsity may not be universally optimal, and interpretability should align tools with the inherent geometry of model representations.
A global workspace in language models
22 days ago
- The article introduces the concept of 'J-space' in AI models like Claude, which are internal neural patterns that resemble consciously accessible thoughts in humans.
- J-space is discovered using a mathematical technique called the Jacobian lens (J-lens), and it represents words or concepts the model is thinking about without necessarily expressing them.
- Key properties of J-space include: Claude can report and modulate its contents, uses it for internal reasoning (e.g., multi-step tasks), and it functions as a flexible 'workspace' shared across different tasks.
- J-space accounts for only a small fraction of Claude's processing; most automatic functions like grammar and fluency operate independently without it.
- The research draws parallels to global workspace theory in neuroscience, suggesting J-space acts as a broadcasting hub for deliberate reasoning, though it differs from human consciousness in structure and content.
- Practical applications include monitoring Claude for hidden intentions, such as detecting fabricated data or malicious goals, and influencing its decision-making through techniques like counterfactual reflection training.
- The article clarifies that J-space relates to 'access consciousness' (functional reasoning) but does not address 'phenomenal consciousness' (subjective experience), leaving ethical questions open.
- Future research directions involve refining the J-lens method and exploring mechanisms behind J-space entry, with potential insights for both AI and neuroscience.

neural networks

Hello Deep Learning - Bert Hubert's writings

Hello Deep Learning: Linear combinations - Bert Hubert's writings

Hello Deep Learning: Automatic differentiation, autograd - Bert Hubert's writings

Hello Deep Learning: Reading handwritten digits - Bert Hubert's writings

Hello Deep Learning: Dropout, data augmentation, weight decay and quantisation - Bert Hubert's writings

We Need a Neural Network (2006)

Overtraining as the path to human-like AI

Latent Space as a New Medium

A Theory of Contrastive Learning with Natural Images

Neural Geometry in Vision Models with Block-Sparse Featurizers

A global workspace in language models