Hasty Briefsbeta

All tags

#machine-learning

166 stories total

Bilingual

The Principles of Diffusion Models
6 months ago
- Diffusion models are based on gradually corrupting data into noise and then learning to reverse this process.
- Three main views of diffusion models: variational (noise removal), score-based (gradient learning), and flow-based (velocity field).
- The common backbone is a time-dependent velocity field that transforms noise into data via a continuous trajectory.
- Applications include controllable generation, efficient numerical solvers, and flow-map models for direct mappings.
- The monograph provides a conceptual and mathematically grounded understanding for those with basic deep-learning knowledge.
Weave (YC W25) is hiring a founding ML engineer
6 months ago
- Weave is building software for engineering teams to enhance productivity and speed.
- The company is well-funded, backed by top investors, and growing rapidly.
- The role involves working directly with the CTO (Andrew) and CEO (Adam).
- Ideal candidates are formidable, pragmatic engineers with experience in shipping ML systems end-to-end.
- Candidates must be empathetic, excellent communicators, and gritty.
- Tech stack includes React + TypeScript (frontend), Go (backend), and Python for ML.
- The founding ML engineer will build ML systems to understand and improve software engineering work.
- The goal is to delight customers by making their jobs 10x easier with intelligent solutions.
Modern Optimizers – An Alchemist's Notes on Deep Learning
6 months ago
- Modern optimizers like Adam are the backbone of modern learning, but spectral-whitening methods claim to outperform Adam.
- Gradient descent traditionally uses a Euclidean distance metric, but non-Euclidean metrics can improve optimization by accounting for parameter sensitivities.
- The whitening metric is derived from the square-root of the Gauss-Newton matrix, offering a conservative estimate for optimization.
- Natural gradient descent uses the Fisher information matrix, which is related to the whitening metric and ensures parameterization-invariant optimization.
- Spectral norm descent relates to the whitening metric by projecting gradients onto orthogonal matrices, optimizing the maximum singular value.
- Optimizers like Adam/RMSProp, Shampoo/SOAP/SPlus, PSGD, and Muon implement spectral-whitening methods with varying computational efficiencies.
- Benchmarking shows that spectral-whitening optimizers like SOAP and Muon outperform Adam in terms of validation loss and steps-to-Adam ratio.
- SOAP is the most effective per gradient-step, while Muon offers computational efficiency, suggesting a potential hybrid approach could be optimal.
- No current optimization method reliably surpasses spectral-whitening optimizers, indicating a need for further innovation in optimization techniques.
Continuous Autoregressive Language Models
6 months ago
- Introduces Continuous Autoregressive Language Models (CALM) for more efficient language generation.
- Shifts from discrete next-token prediction to continuous next-vector prediction.
- Uses a high-fidelity autoencoder to compress chunks of tokens into continuous vectors.
- Reduces generative steps by a factor of K, improving computational efficiency.
- Develops a likelihood-free framework for training, evaluation, and controllable sampling.
- Demonstrates significant performance-compute trade-off improvements over discrete baselines.
- Establishes next-vector prediction as a scalable pathway for ultra-efficient language models.
Honda: 2 years of ml vs 1 month of prompting - heres what we learned
6 months ago
- Major automakers face costly recalls, prompting the creation of an analytics department to categorize warranty claims.
- Traditional SQL queries struggle with modern warranty claim semantics, leading to misclassification and inefficiency.
- A 2023 initiative to automate warranty classification using supervised models faced challenges in data collection, preprocessing, and deployment.
- Large language models (LLMs) were revisited as a solution, showing promise in few-shot performance and cost-effectiveness.
- After iterative prompt tuning, Nova Lite matched or surpassed the performance of supervised models in most categories.
- The shift to LLMs represents a fundamental change in classifier development, reducing dependency on labeled data and annotation cycles.
- Supervised models remain viable for stable, large-scale datasets, but LLMs excel in dynamic or data-scarce environments.
The 1B Token Challenge: Finding the Perfect Pre-Training Mix
6 months ago
- Achieved 90%+ performance of GPT-2 with only 1/10th the training data (1B tokens vs 10B tokens).
- Optimal dataset composition found: 50% finePDFs, 30% DCLM-baseline, 20% FineWeb-Edu.
- Static mixing outperformed curriculum learning, avoiding catastrophic failures and being faster to train.
- Key insights: validation-generalization tradeoff, hard cutoff catastrophes, and the importance of diversity.
- Trained GPT-2-70M model with 70M parameters, achieving comparable performance to original GPT-2.
- Model benchmarks showed near-identical performance with significant cost savings (~50x cheaper).
- Released pre-training dataset collection and trained model for community use.
EyesOff: How I built a screen contact detection model
6 months ago
- The author built a custom model named EyesOff to detect when someone is looking at your screen to prevent shoulder surfing.
- Due to the lack of an existing dataset, the author hand-labeled over 20,000 images and created synthetic gaze labels for pre-training.
- Initial approaches like Eye-Contact-CNN, MediaPipe, and gaze detection models were considered but deemed unsuitable for various reasons.
- The author developed a two-stage training approach: Phase 1 involved pre-training on a gaze regression task, and Phase 2 fine-tuned the model for screen contact classification.
- A key challenge was the lack of data, leading the author to create a custom dataset from sources like the Video Conferencing Dataset (VCD) and YouTube videos.
- The final model achieved ~71% accuracy across close and mid-range distances, with the best performance coming from models trained on expanded datasets.
- Future improvements include collecting more data, reducing model size, and potentially using a dual-model approach for close and far-range detection.
Heretic: Automatic censorship removal for language models
6 months ago
- Heretic is a tool for removing censorship from transformer-based language models without expensive post-training.
- It uses directional ablation (abliteration) combined with a TPE-based parameter optimizer powered by Optuna.
- Heretic works automatically by minimizing refusals and KL divergence from the original model.
- It supports most dense models, including multimodal models and some MoE architectures, but not SSMs/hybrid models.
- Heretic benchmarks the system to optimize batch size and can decensor models like Llama-3.1-8B in about 45 minutes on an RTX 3090.
- After decensoring, users can save the model, upload it to Hugging Face, or chat with it for testing.
- Heretic implements a parametrized variant of directional ablation, orthogonalizing matrices with respect to refusal directions.
- Key innovations include flexible ablation weight kernels, float refusal direction indices, and separate parameters for each component.
- Heretic was developed from scratch, informed by various abliteration techniques and articles.
- The tool is released under the GNU Affero General Public License.
Hachi: An Image Search Engine
5 months ago
- Hachi is a self-hosted search engine designed to search personal data across various devices and storage locations.
- Current implementation focuses on image search, with future plans to support video, text, and audio.
- The project emphasizes minimal external dependencies, hackability, and experimentation with semantic search.
- Features include face recognition, meta-data indexing, and vector search for semantic queries.
- Backend is written in Python and Nim, with a frontend built using HTML, JS, and CSS.
- The system supports distributed queries and aims for high performance on diverse hardware.
- Machine learning models are integrated for tasks like face clustering and semantic embedding.
- The project is tested on Windows and Linux, with potential for extension to other platforms.
- Philosophy includes learning from open-source, optimizing for personal use, and resisting data duplication.
- Future goals include supporting remote storage indexing and improving user interfaces.
The End of the Train-Test Split
5 months ago
- Building a butt classification model at Facebook involves training a CNN for edge detection with high precision and recall.
- Policy team requests a more context-aware model for sexually suggestive content, leading to challenges with LLM decision trees and lower accuracy.
- Discrepancies in labels and policy ambiguity cause issues, with outsourced labelers struggling with nuanced definitions like 'sexually suggestive'.
- Expert input is crucial for complex tasks, but their limited availability makes large labeled datasets difficult to maintain.
- LLMs require clear natural language rules and examples rather than traditional training sets, shifting focus from hyperparameter tuning to policy alignment.
- High error rates in 'golden sets' and expert disagreement highlight the need for continuous feedback loops between policy and engineering teams.
- Traditional train-test splits fail for complex LLM tasks due to label ambiguity and the need for expert review of model explanations.
- Shadow mode testing and direct communication between teams are essential for resolving edge cases and improving model accuracy.
- LLMs excel at enforcing natural language rules but require rigorous policy alignment and ongoing evaluation to handle complex classifications.
- The future of LLMs in tasks like legal or content moderation depends on solving alignment challenges and improving model self-awareness of errors.
NeurIPS best paper awards 2025
5 months ago
- NeurIPS 2025 announced seven groundbreaking papers for Best Paper and Runner-Up awards.
- Awards span topics like diffusion model theory, self-supervised RL, attention mechanisms, and benchmarking methodologies.
- Key papers include 'Artificial Hivemind' on LM diversity, 'Gated Attention' for LLMs, and '1000 Layer Networks' for self-supervised RL.
- Other awarded works explore diffusion model memorization, RL's impact on LLM reasoning, optimal mistake bounds in online learning, and neural scaling laws.
- The selection committee comprised leading researchers across ML topics, nominated by program chairs and approved by general chairs.
Guidance: A cheat code for diffusion models
5 months ago
- Classifier-free diffusion guidance improves conditional diffusion model samples effectively and simply.
- Diffusion models model high-dimensional data distributions by predicting the score function.
- Conditional diffusion models use additional inputs to model conditional distributions.
- Classifier guidance uses a classifier to condition diffusion models but has limitations.
- Classifier-free guidance avoids training a separate classifier by using conditioning dropout.
- Guidance enhances conditioning signal adherence and sample quality at the cost of diversity.
- Guidance can be applied to autoregressive models but differs in effect compared to diffusion models.
- Temperature tuning in diffusion models is less effective than in autoregressive models.
- Guidance is a key factor in the recent success of diffusion models in image generation.
The Universal Weight Subspace Hypothesis
5 months ago
- Deep neural networks trained across diverse tasks exhibit similar low-dimensional parametric subspaces.
- Empirical evidence shows neural networks converge to shared spectral subspaces regardless of initialization, task, or domain.
- Mode-wise spectral analysis of over 1100 models identifies universal subspaces capturing majority variance in few principal directions.
- Spectral decomposition techniques reveal sparse, joint subspaces consistently exploited across diverse tasks and datasets.
- Findings offer insights into intrinsic organization of information within deep networks and implications for model reusability and efficiency.
- Potential to reduce carbon footprint of large-scale neural models through training and inference-efficient algorithms.
The Well: 15TB of Physics Simulations
5 months ago
- The Well is a large-scale collection of machine learning datasets containing numerical simulations of spatiotemporal physical systems.
- It provides 15TB of data across 16 datasets covering diverse domains like biological systems, fluid dynamics, and magneto-hydrodynamic simulations.
- Datasets can be used individually or as part of a broader benchmark suite for machine learning and computational sciences research.
- Installation can be done via PyPI or from source, with options for different hardware accelerations like CUDA.
- Datasets range from 6.9GB to 5.1TB each, requiring significant disk space.
- Data can be downloaded locally or streamed from Hugging Face, with recommendations for local downloads for better performance.
- Benchmarking tools are included, with state-of-the-art models implemented for surrogate modeling.
- Pre-trained model checkpoints are available on Hugging Face for easy loading and use.
- The project is a collaboration led by Polymathic AI with contributions from various institutions worldwide.
- Users are encouraged to cite the project in their research and can contact the maintainers for questions or issues.
Show HN: Cordon – Reduce large log files to anomalous sections
5 months ago
- Cordon uses transformer-based embeddings and density-based scoring for semantic anomaly detection in log files.
- Key principle: Repetitive patterns are considered normal; unusual, rare, or clustered events are highlighted.
- Features include semantic analysis, density-based scoring, noise reduction, and multiple backends (sentence-transformers or llama.cpp).
- GPU acceleration requires NVIDIA GPUs with Pascal architecture or newer; CPU mode is always available.
- Installation options include pip, uv, and cloning the repository for development.
- Basic usage involves running Cordon on log files with options for window size, k-neighbors, and anomaly percentile.
- Advanced configurations allow for GPU acceleration, anomaly range filtering, and detailed output.
- Cordon reduces large log files to semantically significant sections, achieving up to 98% reduction in some cases.
- Workflow includes ingestion, segmentation, vectorization, scoring, thresholding, merging, and formatting.
- Parameters like window_size, k_neighbors, and anomaly_percentile can be adjusted for different log types.
- Use cases include LLM pre-processing, initial triage, anomaly detection, and exploratory analysis.
- GPU acceleration provides significant speedups for large log files, with PyTorch used for k-NN scoring.
AIsbom – open-source CLI to detect "Pickle Bombs" in PyTorch models
5 months ago
- AIsbom is a specialized security and compliance scanner for Machine Learning artifacts.
- Performs Deep Binary Introspection on model files (.pt, .pkl, .safetensors) to detect malware risks and legal license violations.
- Install via PyPI: `pip install aisbom-cli`.
- Scan ML projects with `aisbom scan ./my-project-folder`.
- Generates a combined Security & Legal risk assessment in the terminal.
- Produces a compliant sbom.json (CycloneDX v1.6) with SHA256 hashes and license data.
- Offers an offline viewer at aisbom.io/viewer.html for visualizing risks and compliance stats.
- Detects security risks like RCE in PyTorch files and legal risks like restrictive licenses.
- Includes features like Deep Introspection, Pickle Bomb Detector, License Radar, and Compliance Ready outputs.
- Allows generating test artifacts for verification with `aisbom generate-test-artifacts`.
- Can be integrated into CI/CD pipelines to block unsafe models before merging.
Letta Code
5 months ago
- Letta Code is a memory-first coding agent designed for long-lived agents that persist across sessions and improve with use.
- It focuses on continual learning and memory for coding agents, accumulating context and memory over time to enhance performance.
- Memory initialization can be triggered with the `/init` command, allowing the agent to learn about existing projects through deep research.
- Skill learning enables agents to remember and apply patterns for repeated tasks, improving future task performance.
- Persisted state allows agents to lookup past conversations and messages through the Letta API, supporting vector, full-text, and hybrid search.
- Letta Code is the #1 model-agnostic OSS coding harness on TerminalBench, performing comparably to provider-specific harnesses.
- It can be installed via npm or from source and used with the Letta Developer Platform or a self-hosted Letta server.
Show HN: Misata – synthetic data engine using LLM and Vectorized NumPy
5 months ago
- Misata allows generating realistic multi-table datasets from natural language descriptions without needing to write schemas or provide training data.
- It supports features like auto schema generation, relational integrity, business constraints, and streaming for large datasets (10M+ rows).
- Installation is simple via pip, and it supports multiple LLM providers like Groq, OpenAI, and Ollama.
- Users can generate data for various scenarios like SaaS, e-commerce, and fitness apps with customizable options.
- Misata includes advanced features like noise injection, custom distributions, and conditional values for realistic data generation.
- Performance metrics show high-speed data generation, capable of handling millions of rows efficiently.
- It offers a browser-based trial and enterprise support for complex scenarios.
- Developed by Muhammed Rasin under MIT License.
Show HN: Autograd.c – a tiny ML framework built from scratch
5 months ago
- A minimal reverse mode autograd engine in C
- Features reference counted tensors and arena allocated function nodes
- Includes explicit dependency counting and centralized gradient accumulation
- Supports scalar loss backpropagation
- Small set of core tensor ops with tightly coupled forward and backward code
- Demonstrated with CIFAR-10 binary dataset training
- Training shows progress with loss and accuracy metrics
- Project described as 'tiny torch, but close to metal'
TurboDiffusion: 100–200× Acceleration for Video Diffusion Models
4 months ago
- TurboDiffusion accelerates video diffusion models by 100-200 times using SageAttention, SLA, and rCM.
- Supports models like TurboWan2.1 and TurboWan2.2 with resolutions up to 720p.
- Install via pip or compile from source; requires Python 3.9+ and torch 2.7.0+.
- Different checkpoints available for GPUs with varying memory capacities (quantized for RTX 5090, unquantized for H100).
- Includes detailed inference scripts for T2V and I2V models with customizable parameters.
- Training code provided with support for FSDP2, Ulysses CP, and SAC.
- Community contributions welcomed, with ongoing improvements and integrations planned.
- Citations provided for related research papers and software.

first prev5next

About|Login

#machine-learning

The Principles of Diffusion Models

Weave (YC W25) is hiring a founding ML engineer

Modern Optimizers – An Alchemist's Notes on Deep Learning

Continuous Autoregressive Language Models

Honda: 2 years of ml vs 1 month of prompting - heres what we learned

The 1B Token Challenge: Finding the Perfect Pre-Training Mix

EyesOff: How I built a screen contact detection model

Heretic: Automatic censorship removal for language models

Hachi: An Image Search Engine

The End of the Train-Test Split

NeurIPS best paper awards 2025

Guidance: A cheat code for diffusion models

The Universal Weight Subspace Hypothesis

The Well: 15TB of Physics Simulations

Show HN: Cordon – Reduce large log files to anomalous sections

AIsbom – open-source CLI to detect "Pickle Bombs" in PyTorch models

Letta Code

Show HN: Misata – synthetic data engine using LLM and Vectorized NumPy

Show HN: Autograd.c – a tiny ML framework built from scratch

TurboDiffusion: 100–200× Acceleration for Video Diffusion Models