Hasty Briefsbeta

All tags

#machine-learning

166 stories total

Bilingual

Researchers Uncover Hidden Ingredients Behind AI Creativity
10 months ago
- AI systems like diffusion models (e.g., DALL·E, Imagen, Stable Diffusion) exhibit unexpected creativity by blending elements to create new, coherent images.
- Researchers discovered that technical imperfections in the denoising process (locality and equivariance) drive AI creativity, not just replication.
- Mason Kamb and Surya Ganguli developed the Equivariant Local Score (ELS) machine, which predicts denoised images with 90% accuracy, matching trained diffusion models.
- The study suggests AI creativity arises from focusing on local patches without broader context, similar to how human creativity may fill knowledge gaps.
- Experts note that while the paper explains diffusion model creativity, other AI systems (e.g., large language models) may rely on different mechanisms.
Will Scaling Solve Robotics?
10 months ago
- CoRL 2023 was the largest yet with over 900 attendees, 11 workshops, and nearly 200 accepted papers.
- A central debate: Can training large neural networks on vast datasets solve robotics?
- Foundation models' success in CV/NLP suggests potential for robotics, but challenges remain.
- Arguments for scaling in robotics:
- - Success in CV/NLP with large models and datasets.
- - Early evidence from RT-X, RT-2, and Diffusion Policies papers.
- - Leveraging progress in data, compute, and foundation models.
- - Discovering a simpler manifold of practical robotics tasks.
- - Large models may enable 'common sense' reasoning for robotics.
- Arguments against scaling in robotics:
- - Lack of large-scale robotics data compared to CV/NLP.
- - Diversity in robot embodiments complicates data collection.
- - High variance in environments robots must operate in.
- - High cost and energy consumption of training large models.
- - The '99.X% problem'—real-world applications require near-perfect reliability.
- - Long-horizon tasks compound errors over time.
- - Self-driving car companies' struggles with scaling approaches.
- Misc. related arguments:
- - Learning-based approaches can be deployed robustly despite lack of guarantees.
- - Human-in-the-loop systems as a practical deployment strategy.
- - Using simulators and existing vision/language data to mitigate data scarcity.
- - Combining classical and learning-based approaches for better results.
- Key takeaways:
- - Pursue scaling in robotics but explore other directions too.
- - Focus on real-world mobile manipulation and user-friendly systems.
- - Report negative results to avoid repeated efforts.
- - Encourage innovative, out-of-the-box thinking for new solutions.
Just Ask for Generalization
10 months ago
- Generalizing to desired outcomes may be easier than direct optimization.
- Large, diverse datasets are crucial for generalization in machine learning.
- Deep neural networks excel at absorbing vast amounts of data quickly.
- Overparameterized models can generalize well even after minimizing training loss.
- The 'Grokking' phenomenon shows models can suddenly generalize after prolonged training.
- Memorization is seen as a step towards generalization.
- Language-conditioned models like DALL-E demonstrate impressive generalization capabilities.
- Reinforcement learning struggles with computational efficiency in absorbing diverse data.
- Supervised learning can replace RL by learning a distribution of policies and inferring the best one.
- Techniques like Decision Transformers and Hindsight Language Relabeling leverage generalization.
- Ranking models and data augmentation can infer better-than-demonstrator behavior.
- Meta-reinforcement learning can learn policy improvement operators via supervised learning.
- A 'generalize-and-infer' approach can replace direct optimization in many scenarios.
- Consciousness in AI could be approached through language-conditioned multi-policy models.
- Theory-of-mind behaviors might emerge from models trained on diverse agent interactions.
BharatMLStack – Realtime Inference, MLOps
10 months ago
- BharatMLStack is a production-ready machine learning infrastructure platform aimed at democratizing ML capabilities.
- Mission: Provide scalable, accessible ML solutions for organizations to build, deploy, and manage ML at scale.
- Key Features: Democratize ML, Scale Without Limits, India-First Approach, Real-Time Intelligence, Developer-Friendly.
- Performance Metrics: 1M+ feature vector retrievals/sec, sub-10ms latency, 99.99% uptime, petabyte-scale storage.
- Components: Horizon (Control Plane), Trufflebox UI (ML Management Console), Online Feature Store, Go & Python SDKs.
- Online Feature Store: Sub-10ms feature retrieval, streaming ingestion, backward-compatible versioning, multi-source integration.
- Deployment: Cloud-agnostic, Kubernetes-native, built-in observability, Docker setup available.
- Quick Start: Includes Docker setup, sample data, health checks, and step-by-step tutorials.
- Community: Open-source contributions welcome, with Discord, GitHub Issues, and email support available.
- License: BharatMLStack Business Source License 1.1.
An analytic theory of creativity in convolutional diffusion models
10 months ago
- The paper presents an analytic theory of creativity in convolutional diffusion models.
- Identifies two inductive biases, locality and equivariance, that enable combinatorial creativity.
- Introduces local score (LS) and equivariant local score (ELS) machines for mechanistic interpretation.
- Demonstrates high predictive accuracy (median r² ~0.95) on datasets like CIFAR10 and CelebA.
- Reveals a patch mosaic mechanism where models mix local training patches to create novel images.
- Partially predicts outputs of self-attention UNets (median r² ~0.77), highlighting attention's role in semantic coherence.
Computational Complexity of Neural Networks
9 months ago
- The text discusses the computational complexity of neural networks, focusing on the separation between training (backpropagation) and inference (forward propagation) phases.
- Forward propagation in a feed-forward neural network involves matrix multiplication and activation functions, with a time complexity of O(n^4) under certain assumptions.
- Backpropagation is analyzed to have a higher complexity of O(n^5), making the training phase significantly slower than inference.
- The use of GPUs for parallel execution is highlighted as a method to speed up neural network operations, especially during training.
- Theoretical aspects of learning representations, including the concepts of realizable hypotheses and improper learning, are briefly touched upon.
- The conclusion emphasizes the efficiency of separating training and inference phases due to the higher computational cost of backpropagation.
Machine Learning. Literally.
9 months ago
- Branch predictors are the most successful application of machine learning, achieving over 99% accuracy in practice.
- They are built into modern CPUs to predict program behavior, improving latency and throughput.
- Branch predictors solve a reinforcement learning problem within a Markov decision process framework.
- They use domain-specific knowledge and optimizations, making them barely recognizable to statisticians.
- Target prediction involves predicting if an instruction is a jump, if it will be taken, and its destination address.
- Outcome prediction uses finite state machines (FSMs) and correlating predictors to handle temporal and spatial patterns.
- Advanced predictors use hierarchical architectures and ensembles, like tournament predictors, to improve accuracy.
- Specialized predictors handle loops and function returns efficiently, such as the Subroutine Return Stack for return addresses.
Diffusion Beats Autoregressive in Data-Constrained Settings
9 months ago
- Autoregressive (AR) models have traditionally dominated large language models.
- Diffusion-based language models are emerging as a promising alternative to AR models.
- Diffusion models outperform AR models in data-constrained settings where compute is abundant but data is scarce.
- Masked diffusion models achieve lower validation loss and better downstream performance by leveraging repeated data more effectively.
- Diffusion models benefit from implicit data augmentation due to diverse token orderings and prediction tasks.
- New scaling laws for diffusion models are identified, with a critical compute threshold derived for when diffusion outperforms AR.
- When data is the bottleneck, diffusion models present a compelling alternative to AR models.
ICML 2025 Outstanding Paper Awards
9 months ago
- Batch Normalization addresses internal covariate shift by normalizing layer inputs, enabling higher learning rates and reducing the need for careful initialization.
- Variational inference is enhanced by normalizing flows, allowing for flexible and complex approximate posterior distributions.
- Trust Region Policy Optimization (TRPO) provides guaranteed monotonic improvement in optimizing control policies, effective for large nonlinear policies like neural networks.
- Algorithmic creativity tasks highlight the limitations of next-token learning and the benefits of multi-token approaches like teacherless training and diffusion models.
- Masked diffusion models (MDMs) offer flexibility at inference time but face computational challenges during training; adaptive decoding strategies can significantly improve performance.
- Bayesian quadrature provides interpretable guarantees for predictive models, addressing the limitations of frequentist methods like conformal prediction.
- Score matching is adapted for incomplete data, with importance weighting and variational approaches showing strengths in different settings.
- Machine learning in government programs focuses on equity, with prediction systems aiding in identifying vulnerable individuals.
- CollabLLM enhances human-LLM collaboration by estimating long-term contributions of responses, improving task performance and interactivity.
- AI safety efforts should expand to include impacts on the future of work, advocating for pro-worker frameworks and fair compensation mechanisms.
- The peer review system in AI conferences needs reform to include bi-directional feedback and systematic rewards for reviewers to improve quality and accountability.
Aging Clock Unveils Compounds That Rejuvenate Brain Cells
9 months ago
- Scientists developed a computational 'aging clock' to measure the biological age of brain cells.
- The aging clock uses gene expression data from 365 genes to predict biological age.
- Identified 453 compounds with potential to reverse age-related decline in brain cells.
- Initial tests on mice showed improvements in behavior, cognition, and molecular markers of youth.
- The platform can help discover new neuroprotective and rejuvenating interventions.
- Some identified compounds are already used to treat neurological disorders.
- The study provides a foundation for future therapeutic development against neurodegeneration.
Unsloth Dynamic 2.0 GGUFs
9 months ago
- Unsloth introduces Dynamic v2.0 quantization, a major upgrade improving accuracy and efficiency for quantized LLMs.
- Dynamic v2.0 features revamped layer selection, model-specific quants, and new formats like Q4_NL and Q5.1 for better performance on Apple Silicon and ARM devices.
- The method includes a high-quality calibration dataset with over 1.5M tokens to enhance conversational chat performance.
- Unsloth collaborates with major AI teams (Meta, Google, Microsoft) to fix critical bugs, boosting model accuracy.
- Dynamic v2.0 now works on all models, including MoEs and non-MoEs, unlike its predecessor which was limited to MoE architectures.
- Benchmarking includes a new efficiency metric considering disk size and MMLU scores, with Dynamic 2.0 showing superior performance.
- KL Divergence is used as a key metric to measure quantization accuracy, aiming to minimize flips in model answers.
- Unsloth addresses calibration dataset overfitting by using diverse datasets beyond Wikipedia for fair testing.
- Replicating MMLU 5-shot results was challenging due to subtle implementation issues, leading to custom benchmarking solutions.
- Gemma 3 QAT benchmarks show impressive results, with Dynamic 2.0 versions offering smaller sizes and higher accuracy.
- Bug fixes for Llama 4 include RoPE scaling adjustments and QK Norm epsilon corrections, significantly improving MMLU scores.
- Instructions provided for running Llama 4 Scout using llama.cpp, showcasing Unsloth's Dynamic 2.0 quantizations.
Show HN: Cant, rust nn lib for learning
9 months ago
- A machine learning library written in Rust with PyTorch-like functionality.
- Features include automatic differentiation, tensor operations, and GPU acceleration.
- Supports CPU operations using ndarray and Metal GPU acceleration for macOS.
- Includes a PyTorch-like API for ease of use.
- Project setup includes commands for testing, building, formatting, and coverage reports.
- Key components: Tensor struct, computation graph manager, operation definitions, and shape handling.
- Backend support includes CPU (ndarray) and Metal GPU (macOS).
- Testing includes unit tests, model tests, and operation verification.
- Licensed under MIT License.
Google is experimenting with machine-learning powered age estimation tech in US
9 months ago
- Google is testing machine-learning tech in the U.S. to estimate user age and filter content across its products.
- Under-18 users will receive an email about changes, including disabled Maps timeline, no personalized ads, and restricted content.
- YouTube will enable digital wellbeing features like break reminders and bedtime notifications for younger users.
- Users can appeal age misclassification by submitting a government ID or selfie.
- Google has tested age estimation in some markets and is expanding it to all U.S. Google accounts.
- Other platforms like Instagram and Roblox are also using AI for age estimation.
- Lawmakers are pushing for rules on age-appropriate content, with some U.S. states and the UK implementing age verification laws.
Break the quadratic wall of Transformer attention: WERSA, paper+code open source
9 months ago
- WERSA is a novel attention mechanism with linear O(n) time complexity, designed for scaling Transformer models to very long sequences.
- It combines multi-resolution analysis (Haar wavelet transforms), adaptive filtering (MLP-generated filters), and random feature projection for linear complexity.
- Installation requires PyTorch, Hugging Face Transformers, and the WERSA package from the repository.
- Quickstart examples show how to build Qwen-like models with WERSA, including 8B and 0.6B parameter configurations.
- Includes training scripts for pre-training models from scratch and testing generation capabilities.
- The project is licensed under Apache License 2.0 and the paper is available on arXiv.
Show HN: Building a web search engine from scratch with 3B neural embeddings
9 months ago
- The author built a web search engine from scratch in two months, focusing on improving search quality and relevance using neural embeddings.
- Key motivations included addressing the decline in search engine quality due to SEO spam and leveraging transformer-based text embedding models for better natural language comprehension.
- The search engine was designed to understand intent rather than keywords, enabling it to handle complex, nuanced queries effectively.
- Technical highlights include generating 3 billion SBERT embeddings with 200 GPUs, crawling 50K pages per second, and achieving end-to-end query latency of 500 ms.
- The system architecture involved components like normalization, chunking, a crawler, pipeline, storage, service mesh, GPU buildout, sharded HNSW, and knowledge graph integration.
- The search engine demonstrated superior performance in understanding and answering complex queries compared to traditional keyword-based search engines.
- Cost optimization was a significant focus, with the author leveraging lesser-known, cost-efficient services to manage expenses.
- The project concluded with insights into the potential of neural embeddings for search and plans for future improvements, including better quality filtering and leveraging LLMs for reranking.
Writing an LLM from scratch, part 17 – the feed-forward network
9 months ago
- The feed-forward network in LLMs is crucial for processing context vectors after attention mechanisms.
- It consists of two linear layers with a GELU activation function, expanding and then reducing dimensions.
- Attention mechanisms gather information, but feed-forward networks perform the 'thinking' or pattern-matching.
- Feed-forward networks contain more parameters than attention mechanisms, indicating their importance.
- A single hidden layer in the feed-forward network acts as a universal approximator.
- The author initially underestimated the role of feed-forward networks in LLMs.
- Future posts may explore deeper networks and related research papers.
Do we understand how neural networks work?
9 months ago
- Neural networks are fundamentally made of matrices, a well-understood mathematical concept.
- Training neural networks involves gradient descent, a calculus-based optimization method.
- The objective of training (e.g., predicting the next token in LLMs) is clearly defined but leads to complex outcomes.
- Despite understanding the training process, the end results (learned statistics) are complex and not fully understood.
- LLMs are essentially advanced autocomplete systems, bundling statistics about language or images.
- Mechanistic interpretability is a subfield attempting to reverse-engineer how neural networks function internally.
- Examples like 'Golden Gate Claude' show limited but precise understanding of specific neural network features.
- LLMs can perform tasks like arithmetic by developing internal, non-human-like methods.
- Understanding neural networks is more art than science at the edge of current knowledge.
- Practical use of neural networks doesn't require deep understanding, but research and safety do.
Dyna – Logic Programming for Machine Learning
9 months ago
- Dyna is a programming language designed for machine learning researchers, building on logic programming languages like Datalog and Prolog.
- Dyna allows flexible execution orders and weighted rules, enabling complex programs to be written concisely.
- Example Dyna programs include matrix multiplication, the Fibonacci sequence, CKY parsing, and neural networks.
- The Dyna project started in 2004 to bridge the gap between mathematical notation and executable code for ML researchers.
- Dyna 1.0 extended Datalog by allowing any semiring, making it useful for dynamic programming.
- Dyna 2.0 introduced functions, free variables, lazy expressions, and prototype-based inheritance (dynabases).
- Ongoing research includes using relational algebra and term rewriting for implementation and reinforcement learning for optimization.
- Several research papers and theses have been published on Dyna, covering its development and applications.
- Multiple implementations of Dyna exist, including Dyna3 (Clojure), Dyna-R (Python), Dyna-Pi (Python), Dyna-Phi (Truffle/Graal), and earlier versions in Haskell and Python.
Show HN: I built a toy TPU that can do inference and training on the XOR problem
9 months ago
- The project aimed to build a 'toy' TPU (Tensor Processing Unit) from scratch, focusing on understanding and replicating its functionality for both inference and training.
- The team chose to build a TPU because it was a challenging project with no well-documented open-source ML accelerators that handled both inference and training.
- Design philosophy: 'ALWAYS TRY THE HACKY WAY'—prioritizing original ideas before consulting external sources to reinvent rather than reverse-engineer the TPU.
- The TPU is an ASIC designed by Google for efficient ML workloads, excelling at matrix multiplications, which dominate deep learning computations.
- Key hardware concepts: clock cycles, Verilog for hardware description, and systolic arrays for efficient matrix multiplication.
- The systolic array consists of Processing Elements (PEs) that perform multiply-accumulate operations, enabling parallel computation.
- The XOR problem was used as a simple case study to demonstrate inference and training, requiring an MLP due to its non-linear decision boundaries.
- Pipelining and double buffering were implemented to optimize performance, ensuring continuous inference and efficient weight updates.
- Backpropagation was integrated into the design, leveraging the systolic array for gradient calculations and weight updates.
- A unified buffer (UB) was introduced to store intermediate values during training, improving efficiency and scalability.
- The final design included a custom instruction set (ISA) and control unit to manage operations, achieving a fully functional 'toy' TPU.
A Short Introduction to Optimal Transport and Wasserstein Distance
8 months ago
- Optimal transport theory provides a method to measure distances between probability distributions, addressing limitations of KL divergence.
- The Wasserstein distance, or Earth Mover's Distance, is symmetric and satisfies the triangle inequality, making it a robust metric.
- Optimal transport can be visualized through physical analogies like moving dirt to fill holes, with costs based on distances.
- Discretizing the problem allows solving optimal transport via linear programming, making it computationally tractable.
- Entropy regularization is a key innovation that simplifies computation by penalizing low-entropy transport plans.
- Applications of optimal transport span machine learning, imaging, generative models, and biological data analysis.

first prev2next

About|Login

#machine-learning

Researchers Uncover Hidden Ingredients Behind AI Creativity

Will Scaling Solve Robotics?

Just Ask for Generalization

BharatMLStack – Realtime Inference, MLOps

An analytic theory of creativity in convolutional diffusion models

Computational Complexity of Neural Networks

Machine Learning. Literally.

Diffusion Beats Autoregressive in Data-Constrained Settings

ICML 2025 Outstanding Paper Awards

Aging Clock Unveils Compounds That Rejuvenate Brain Cells

Unsloth Dynamic 2.0 GGUFs

Show HN: Cant, rust nn lib for learning

Google is experimenting with machine-learning powered age estimation tech in US

Break the quadratic wall of Transformer attention: WERSA, paper+code open source

Show HN: Building a web search engine from scratch with 3B neural embeddings

Writing an LLM from scratch, part 17 – the feed-forward network

Do we understand how neural networks work?

Dyna – Logic Programming for Machine Learning

Show HN: I built a toy TPU that can do inference and training on the XOR problem

A Short Introduction to Optimal Transport and Wasserstein Distance