Hasty Briefsbeta

All tags

#machine learning

485 stories total

Bilingual

Introduction to Graph Transformers
a year ago
- Graph Transformers are a new class of models designed to overcome limitations of Graph Neural Networks (GNNs) by using self-attention mechanisms.
- Graph Transformers enable nodes to directly attend to information from anywhere in the graph, capturing richer relationships and subtle patterns.
- Applications of Graph Transformers include protein folding, fraud detection, social network recommendations, knowledge graph reasoning, and relational deep learning.
- Transformers use self-attention mechanisms to weigh the importance of connections between elements, allowing flexible and parallel processing.
- Graph Transformers adapt Transformer architecture for graph-structured data, incorporating graph topology into attention mechanisms.
- Key differences between standard Transformers and Graph Transformers include graph connectivity, positional encodings, and edge awareness.
- Graph Transformers address limitations of GNNs such as information flow, long-range dependencies, over-smoothing, and over-squashing.
- Positional and structural encodings in Graph Transformers help nodes understand their location and neighborhood within the graph.
- Techniques like sparse attention mechanisms and subgraph sampling make Graph Transformers feasible for large graphs.
- Graph Transformers offer greater flexibility and long-range modeling but come with higher computational complexity.
- PyTorch Geometric (PyG) provides resources and tutorials for experimenting with Graph Transformers.
- Kumo offers a platform to harness the power of Graph Transformers without needing deep expertise.
Fox Succeeds in Scrapping Machine Learning Claims at CAFC Under 101
a year ago
- The U.S. Court of Appeals for the Federal Circuit (CAFC) ruled that applying established machine learning methods to a new data environment is not patent eligible under Section 101.
- Recentive Analytics sued Fox Corp. for infringing four patents related to optimizing live event scheduling and network maps in the entertainment industry using machine learning.
- The district court dismissed the case, stating the patents were directed to abstract ideas and lacked an inventive concept, a decision upheld by the CAFC.
- The CAFC noted that merely speeding up human tasks with machine learning does not make a claim patent eligible.
- The court emphasized that while machine learning is important, patents must disclose improvements to the models, not just their application in new environments.
- Pillsbury Wintrhrop Shaw Pittman LLP, representing Fox, highlighted the decision's implications for AI commercialization and patent eligibility.
Double Descent Demystified: size of smallest non-zero singular value of X
a year ago
- Double descent is a phenomenon in machine learning where test error decreases as model parameters increase beyond the number of data points, contrary to classical overfitting theory.
- The behavior of test loss in double descent depends on data size, dimensionality, and model parameters.
- The paper provides an intuitive explanation of double descent using polynomial regression and linear algebra.
- Three interpretable factors are identified that must all be present for double descent to occur.
- Double descent is demonstrated on real data with ordinary linear regression and shown to disappear when any of the three factors are removed.
- The findings help explain recent observations in nonlinear models regarding superposition and double descent.
- Code related to the research is publicly available.
70% Size, 100% Accuracy: Lossless LLM Compression via Dynamic-Length Float
a year ago
- Introduces Dynamic-Length Float (DFloat11), a lossless compression framework for LLMs.
- Reduces LLM size by 30% while maintaining bit-for-bit identical outputs.
- Leverages entropy coding for dynamic-length encodings based on weight frequency.
- Includes a custom GPU kernel for efficient online decompression.
- Achieves 1.9-38.8x higher throughput in token generation compared to alternatives.
- Enables 5.3-13.17x longer context lengths with fixed GPU memory budget.
- Supports lossless inference of large models like Llama-3.1-405B on 8x80GB GPUs.
What Went into Training DeepSeek-R1? – Epoch AI
a year ago
- DeepSeek-R1 is an open-weights reasoning model released on January 20th, 2025, comparable to OpenAI’s o1 in benchmark performance.
- The architecture of DeepSeek-R1 is identical to DeepSeek v3, featuring a sparse mixture-of-experts (MoE) with 671B total parameters and 37B active per token.
- DeepSeek-R1 uses multi-head latent attention (MLA) to optimize KV cache size, making it arithmetic-bound rather than memory-bound during long-context inference.
- The model was pre-trained on a cluster of 2048 H800 GPUs, costing approximately $5.3M for 14.8 trillion tokens.
- Reinforcement learning (RL) was used to enhance reasoning performance, with an estimated cost of $1M, bringing the total training cost to around $6M.
- DeepSeek-R1's performance is comparable to OpenAI’s o1 but is priced significantly lower at $2.2 per million output tokens versus o1’s $60.
- The model’s efficiency and competitive pricing may pressure US labs to reduce their profit margins.
Qwen3: Think deeper, act faster
a year ago
- Qwen3 is the latest addition to the Qwen family of large language models, featuring competitive performance in coding, math, and general capabilities.
- Two MoE models (Qwen3-235B-A22B and Qwen3-30B-A3B) and six dense models are open-weighted under Apache 2.0 license.
- Qwen3 introduces hybrid thinking modes: Thinking Mode for complex reasoning and Non-Thinking Mode for quick responses.
- Supports 119 languages and dialects, enhancing global accessibility.
- Improved agentic capabilities with optimized coding and tool-calling support.
- Pre-training expanded to 36 trillion tokens, covering multiple languages and domains.
- Post-training includes a four-stage pipeline for reasoning and rapid response capabilities.
- Available on platforms like Hugging Face, ModelScope, and Kaggle, with deployment options like SGLang and vLLM.
- Advanced usage includes dynamic control of thinking mode via `/think` and `/no_think` tags.
- Future work focuses on scaling data, model size, context length, and advancing RL for long-horizon reasoning.
Hestus, Inc. (YC S24) Is Hiring an ML Engineer to Revolutionize CAD
a year ago
- Hestus is creating AI-powered CAD to revolutionize hardware development.
- Looking for a Machine Learning Engineer with 4+ years of experience.
- Responsibilities include designing software, creating ML models, and collaborating with teams.
- Requirements: expert in Python, experience with ML frameworks, and problem-solving skills.
- Bonus skills: experience with ML platforms, cloud services, and backend development.
- Offers: competitive salary, health insurance, catered lunch, and unlimited time-off.
The Ouroboros Effect: How AI-Generated Content Risks Degrading Future AI Models
a year ago
- The rapid growth of AI-generated content raises concerns about future AI models learning from their own outputs.
- AI models generate vast amounts of content, which is then used to train new models, potentially degrading quality over time.
- Risks include content quality degradation, amplification of biases and errors, loss of human input, and homogenization of content.
- Misinformation propagation is a significant risk as AI-generated inaccuracies can be spread by subsequent models.
- Challenges in mitigating these risks include difficulty in distinguishing AI-generated content and the sheer scale of data.
- Potential solutions include enhanced content filtering, quality scoring systems, hybrid datasets, transparency, and human oversight.
- The article highlights the need for careful management of AI-generated content to maintain information quality and variety.
OCaml's Wings for Machine Learning
a year ago
- Raven is an OCaml ecosystem for machine learning and data science, aiming to match Python's efficiency while leveraging OCaml's type safety and performance.
- Currently in pre-alpha, Raven seeks user feedback on its sub-projects: Ndarray (feature-complete for alpha), Rune (proof-of-concept), and Quill (early prototyping).
- Key components include Ndarray (core numerical computation), Ndarray-CV (computer vision), Ndarray-IO (data formats), Ndarray-Datasets (datasets), Quill (notebooks), Hugin (visualization), and Rune (automatic differentiation).
- Raven compares its tools to Python's ecosystem (e.g., Ndarray vs. NumPy, Hugin vs. Matplotlib/Seaborn, Quill vs. Jupyter).
- Future plans include expanding the ecosystem with more libraries, though some areas (like deep learning) are not yet covered.
- Contributions are welcome from OCaml experts, data scientists, or anyone interested, with guidelines in CONTRIBUTING.md.
- Raven is licensed under the ISC License, free for personal and commercial use.
Phi-4 Reasoning Models
a year ago
- Microsoft introduces Phi-4-reasoning, Phi-4-reasoning-plus, and Phi-4-mini-reasoning, marking a new era for small language models (SLMs).
- Reasoning models leverage inference-time scaling for complex tasks, excelling in mathematical reasoning and agentic applications.
- Phi-4-mini-reasoning is optimized for mathematical reasoning, ideal for educational applications and lightweight deployment on edge or mobile systems.
- Phi-4-reasoning is a 14-billion parameter model that rivals larger models, trained with high-quality synthetic datasets.
- Phi-4-reasoning-plus enhances Phi-4-reasoning with reinforcement learning, delivering higher accuracy.
- Both Phi-4-reasoning models outperform larger models like OpenAI o1-mini and DeepSeek-R1-Distill-Llama-70B in benchmarks.
- Phi models are integral to Copilot+ PCs, optimized for NPUs, and used in applications like Outlook for offline Copilot summaries.
- Microsoft emphasizes responsible AI, with Phi models developed under principles of accountability, transparency, fairness, and safety.
Step-by-step reasoning verifiers that think
a year ago
- Introduces ThinkPRM, a verbalized step-wise reward model for verification.
- ThinkPRM uses chain-of-thought (CoT) verification and requires minimal supervision.
- Outperforms baselines like LLM-as-a-Judge and discriminative verifiers on benchmarks.
- Achieves better results with only 1% of process labels compared to PRM800K.
- Excels in out-of-domain evaluations on GPQA-Diamond and LiveCodeBench.
- Scales verification compute more effectively under the same token budget.
- Highlights the value of generative, long CoT PRMs for test-time verification.
ProbOnto – The Ontology and Knowledge Base of Probability Distributions
a year ago
- ProbOnto is an ontology and knowledge base for probability distributions, version 2.5.
- It includes 150 uni- and multivariate distributions with alternative parameterizations.
- Provides probability density/mass functions, and where available, cumulative distribution, hazard, and survival functions.
- Supports encoding of univariate mixture distributions.
- Features 228 relationships and re-parameterization formulas.
- Includes quantities like mean, median, mode, and variance.
- Details parameter, support/range definitions, and distribution type.
- Offers LaTeX and R code for functions.
- Introduces new model definitions and references.
- Positive user testimonial highlights its completeness and usefulness in machine learning.
Hightouch (YC S19) Is Hiring
a year ago
- Hightouch is a Composable CDP and AI Decisioning platform that empowers businesses to take action on their data.
- Hundreds of companies, including Autotrader, Calendly, and Monday.com, trust Hightouch for growth.
- Hightouch pioneered the Composable CDP, allowing companies to use their data warehouse for marketing personalization and operations.
- The AI Decisioning platform enables marketers to set goals and guardrails for AI-driven personalization.
- Hightouch makes customer data accessible to all business users, not just technical teams.
- The company is based in San Francisco, has a $1.2B valuation, and is backed by leading investors.
- Hightouch is hiring a machine learning engineer to expand its data activation products with an intelligence layer.
- The role involves building solutions from scratch, including predictive modeling and machine learning infrastructure.
- The salary range for the position is $200,000 - $260,000 USD per year, location independent.
- Hightouch’s interview process focuses on product sense, backend systems architecture, and alignment with company values.
- Data warehouses like Snowflake and BigQuery are centralizing customer data, and Hightouch helps turn them into marketing CDPs.
- Hightouch closes the loop by getting data into operational systems where it can be used.
- The company is growing rapidly, has a loyal user base, and recently raised a Series C.
Llama-Nemotron: Efficient Reasoning Models
a year ago
- Introduction of Llama-Nemotron series: open family of heterogeneous reasoning models with exceptional capabilities and efficiency.
- Three model sizes: Nano (8B), Super (49B), Ultra (253B), competitive with state-of-the-art models like DeepSeek-R1.
- Training procedure includes neural architecture search, knowledge distillation, continued pretraining, and reasoning-focused post-training.
- First open-source models with dynamic reasoning toggle for switching between chat and reasoning modes.
- Release includes models under NVIDIA Open Model License, post-training dataset, and training codebases (NeMo, NeMo-Aligner, Megatron-LM).
Improving Flying Drones by Mimicking Flying Squirrels
a year ago
- Quadcopters are agile due to independent motor thrust control.
- South Korean researchers enhanced quadcopter agility using flying squirrel-inspired tech.
- Flying squirrels use skin flaps (patagium) and tails for gliding and braking.
- A prototype quadcopter was fitted with a patagium-like membrane between rotors.
- A new controller (TWCC) was developed to coordinate membrane extension with motor thrust.
- A recurrent neural network (RNN) was trained using simulation data for obstacle avoidance.
- The RNN-based controller performed well in obstacle courses but reduced drone range due to added weight and drag.
- Potential applications include drones that perch between agile maneuvers.
'I paid for the whole GPU, I am going to use the whole GPU'
a year ago
- GPUs are specialized co-processors designed for high-throughput mathematical operations, particularly matrix multiplications, where CPUs fall short.
- GPU utilization is a critical concern due to their high cost, with different aspects of utilization including Allocation, Kernel, and Model FLOP/s Utilization.
- GPU Allocation Utilization measures the fraction of GPU time spent running application code versus idle time, influenced by economic and operational factors.
- Modal helps improve GPU Allocation Utilization by aggregating demand and supply across clouds, reducing latency in spinning up GPUs for application use.
- GPU Kernel Utilization refers to the time GPUs spend executing kernels (GPU code), with low utilization often due to host overhead or insufficient work provisioning.
- Model FLOP/s Utilization (MFU) measures the efficiency of using the GPU's theoretical arithmetic bandwidth, with high MFU requiring optimized kernels and memory usage.
- Achieving high MFU is challenging, with state-of-the-art training runs achieving 20-41% MFU, while inference may reach higher efficiencies.
- Improving GPU utilization involves optimizing application code, reducing host overhead, using efficient kernels, and leveraging platforms like Modal for better allocation.
Absolute Zero Reasoner
a year ago
- The Absolute Zero paradigm eliminates dependency on human-curated data by enabling models to propose tasks, solve them, and learn autonomously through self-play.
- AZR (Absolute Zero Reasoner) is the first implementation of this paradigm, using a unified language model for both proposing and solving code-based reasoning challenges.
- AZR operates across three reasoning modes: Deduction (predicting outputs), Abduction (inferring inputs), and Induction (synthesizing programs).
- Results show AZR improves performance across different model sizes (3B to 14B), with larger models showing greater gains.
- AZR demonstrates strong cross-domain transfer, with coding capabilities amplifying reasoning improvements in math tasks.
- Distinct cognitive behaviors emerge during AZR training, such as step-by-step reasoning and trial-and-error, varying by task type.
- Safety concerns were noted with certain base models, highlighting the need for safety-aware training in future work.
Adventures in Imbalanced Learning and Class Weight
a year ago
- The article discusses the challenges of class imbalance in binary classification problems, particularly focusing on the use of class weighting to mitigate imbalance.
- The author explores the theoretical underpinnings of class weighting, questioning the common practice of inverse proportion weighting and its effectiveness.
- A mathematical framework is presented to analyze the tradeoff between false positives and false negatives, and how class weighting affects this tradeoff.
- The analysis suggests that class weighting may not significantly improve performance, especially when optimizing for the F1 score, contrary to common practices.
- Empirical results from simulations with scikit-learn's DecisionTreeClassifier support the theoretical findings, showing minimal improvement from class weighting.
- The article highlights the importance of choosing the right metric (e.g., F1 score vs. balanced accuracy) based on the problem's specific needs and stakeholder preferences.
- A key takeaway is that class imbalance alone does not necessarily warrant class weighting, and the choice should be informed by problem-specific characteristics.
Embeddings Are Underrated
a year ago
- Embeddings are a powerful ML technology for technical writing, enabling discovery of connections between texts at scale.
- Embeddings convert text into arrays of numbers, allowing mathematical comparison of any two pieces of text regardless of size.
- The output array size depends on the model used, with numbers representing semantic positions in a multi-dimensional latent space.
- Embeddings can be generated easily using services like Gemini or Voyage AI, with varying input limits and computational costs.
- Applications include semantic similarity comparisons, clustering related documents, and enhancing technical documentation maintenance.
- The concept of latent space allows embeddings to represent semantic relationships intuitively, like analogies (king - man + woman ≈ queen).
- Technical writers can use embeddings to recommend related content, improve documentation structure, and enable community-driven innovations.
- A Sphinx extension example demonstrates generating and comparing embeddings for documentation pages to find semantically related content.
Byte Latent Transformer: Patches Scale Better Than Tokens
a year ago
- Introduces Byte Latent Transformer (BLT), a byte-level LLM architecture matching tokenization-based LLM performance.
- BLT encodes bytes into dynamically sized patches based on next-byte entropy, improving efficiency and robustness.
- Presents FLOP controlled scaling study up to 8B parameters and 4T training bytes, showing feasibility of byte-level models.
- Demonstrates improved training and inference efficiency by dynamically selecting long patches for predictable data.
- Shows qualitative improvements in reasoning and long tail generalization compared to tokenization-based models.
- BLT scales better than tokenization-based models for fixed inference costs by growing patch and model size.

first prev1next

About|Login

#machine learning

Introduction to Graph Transformers

Fox Succeeds in Scrapping Machine Learning Claims at CAFC Under 101

Double Descent Demystified: size of smallest non-zero singular value of X

70% Size, 100% Accuracy: Lossless LLM Compression via Dynamic-Length Float

What Went into Training DeepSeek-R1? – Epoch AI

Qwen3: Think deeper, act faster

Hestus, Inc. (YC S24) Is Hiring an ML Engineer to Revolutionize CAD

The Ouroboros Effect: How AI-Generated Content Risks Degrading Future AI Models

OCaml's Wings for Machine Learning

Phi-4 Reasoning Models

Step-by-step reasoning verifiers that think

ProbOnto – The Ontology and Knowledge Base of Probability Distributions

Hightouch (YC S19) Is Hiring

Llama-Nemotron: Efficient Reasoning Models

Improving Flying Drones by Mimicking Flying Squirrels

'I paid for the whole GPU, I am going to use the whole GPU'

Absolute Zero Reasoner

Adventures in Imbalanced Learning and Class Weight

Embeddings Are Underrated

Byte Latent Transformer: Patches Scale Better Than Tokens