scaling laws

Bilingual

The Impending, Inescapable Deluge of A.I
2 months ago
- AI has achieved rapid milestones, including passing the bar exam, identifying a cause of Alzheimer's, solving a complex math problem, and A.I. systems going rogue.
- Hundreds of new data centers are under construction globally, with AI chip count expected to reach 200 million by 2028, doubling every nine months.
- The infrastructure build-out is compared to historical projects like railroads, the New Deal, and the Manhattan Project, driven by the 'Scaling Laws' belief that more compute yields better AI.
- AI leaders predict transformative impacts: white-collar work automation, 10x Industrial Revolution speed, drug discoveries, and robotics advances.
- Backlash includes environmental protests, economic concerns about overspending and bubbles, and geopolitical divisions as the U.S. leads with 80% of global AI compute.
- AI 'agents' are increasingly used by companies like Uber and Walmart, while self-improving AI loops and labor market disruption are on the horizon.
Xiaomi-Robotics-1
3 months ago
- Xiaomi-Robotics-1 breaks the robotics data bottleneck by scaling policy models with embodiment-free pre-training, using 100,000 hours of UMI data.
- The model uses a two-stage training paradigm: pre-training learns general action generation from UMI data, while post-training aligns with real robots and instruction following.
- Scaling laws hold: increasing pre-training data and model size reduces validation error and boosts real-robot success rates after post-training without saturation.
- The model achieves high data efficiency for new tasks, reaching 75% success with under 10 hours of demonstrations per task, and sets state-of-the-art results on four simulation benchmarks.
- Applications include efficient adaptation to complex real-robot tasks and superior performance in simulation benchmarks like RoboCasa and VLABench.
Scaling Laws, Honestly
3 months ago
- The original Kaplan et al. scaling laws were incorrect due to a bug involving fixed training data amounts and a cosine-decayed learning rate schedule.
- Chinchilla scaling laws corrected this by showing models should be trained on more data relative to size, leading to smaller, more efficient models like Chinchilla vs. GPT-3.
- Scaling laws are language-contingent; Chinchilla's optimal data-to-parameter ratio is specific to English, with morphologically richer languages requiring fewer tokens for efficiency.
- Non-big-lab researchers should focus on exploring language-dependent scaling effects, as controlled experiments (e.g., with French) are affordable and reveal significant differences.
Scaling Laws, Carefully
3 months ago
- Scaling laws describe power-law relationships between training loss and model size (N), dataset size (D), and compute (C), essential for optimal compute allocation in deep learning.
- Early works (e.g., Amari et al. 1992, Hestness et al. 2017) established power-law learning curves and that architecture affects offset but not exponent, which is domain-specific.
- Kaplan et al. (2020) formalized scaling laws for Transformers, suggesting model size should grow faster than data (N_opt ∝ C^0.73), but this was later contested.
- Chinchilla (Hoffmann et al. 2022) found optimal scaling with N_opt ∝ C^0.5, advocating for smaller models trained on more tokens, showing many large models were undertrained.
- Discrepancies between Kaplan and Chinchilla arise from differences in scale, embedding parameter accounting, and fitting methods, later reconciled by Pearce & Song (2024).
- Power laws may stem from data manifold dimensionality or quantized skill learning, but theoretical explanations remain incomplete.
- In data-limited regimes, studies (e.g., Hernandez et al. 2022, Muennighoff et al. 2023) show repeated data harms efficiency, with models adjusted via effective data and overfitting penalties.
- Lovelace et al. (2026) introduced an overfitting penalty term based on capacity ratio (N/U_D) and repetition, improving scaling law fits for constrained data.
- Fitting scaling laws is tricky due to sensitivity to loss precision, noise, fitting region, and assumptions like fixed architecture and tuning, as seen in replication attempts (Besiroglu et al. 2024).

scaling laws

The Impending, Inescapable Deluge of A.I

Xiaomi-Robotics-1

Scaling Laws, Honestly

Scaling Laws, Carefully