Bilingual

June gwern.net newsletter
a day ago
- Gwern.net published three new pages or essays in June 2020.
- Newsletter includes links on CRISPR, population screening, AI scaling, politics, and technological unemployment.
- The canonical edition is available on gwern.net website.
February 2021 Gwern.net Newsletter
a day ago
- Gwern.net newsletter for February 2021 covers AI scaling, genetics, statistics, politics, psychology, technology, economics, and philosophy.
- AI section highlights advancements in neural text generation, language model fine-tuning, prompt programming, and brain-computer interfaces.
- Genetics topics include heritability of traits, recent evolution, and polygenic burden impacts on health and cognition.
- Statistics/meta-science discusses lessons from history, microCOVID risk calculator, and critiques of modern mathematics education.
- Politics/religion covers fashion cycles, lunar pandemic quarantine critique, and Lysenkoism in Russia.
- Psychology/biology focuses on semaglutide for weight loss, bat immune defenses, and placebo effects in psychedelic microdosing.
- Technology section mentions wringing gauge blocks as a mysterious physical phenomenon.
- Economics analyzes renewable energy cost reduction, IQ and trading performance, and genetic endowments in wealth inequality.
- Philosophy questions ethics experts' behavior, with studies showing they may not be more ethical than others.
- Key themes include scaling laws, biological plausibility of backprop, and the role of placebo in psychology.
April 2021 newsletter
a day ago
- Highlights include AI scaling progress, with record-breaking East Asian language models like Naver's 204B-parameter HyperCLOVA and Huawei's 200B-parameter PanGu-α.
- Deep reinforcement learning advancements: MuZero improvements (MuZero Unplugged, Sampled MuZero), Decision Transformer, and scaling laws for board games.
- Other notable AI papers: Perceiver, Z-IL, super-convergence, and carbon emission estimates for large neural networks.
- Genetics: heritability of exercise response, medieval plague DNA analysis, and news on CRISPR bans.
- Technology: Emacs native compilation, SQLite on static hosts, Fontemon (game in a font), and SIGBOVIK 2021 highlights.
Overtraining as the path to human-like AI
10 days ago
- Gwern proposes that overtraining large models on small datasets could lead to grokking, enabling human-like generalization in AI.
- Grokking is a phenomenon where prolonged training forces models to find simpler, deeper understandings of data, beyond mere memorization.
- Current AI labs focus on training smaller models on vast datasets, which may hinder grokking and limit generalization capabilities.
- Gwern suggests training a massive model (e.g., 100 trillion parameters) on a constrained dataset to encourage deeper learning, though this approach is untested and risky.
- The idea faces technical and political challenges, including high costs and the appearance of failure during training until a breakthrough occurs.
Materials innovation has a scale-up problem, not discovery
18 days ago
- Richard Feynman's 1959 'plenty of room at the bottom' speech initiated atomic-scale control, leading to modern electronics and Moore's Law.
- Current materials for AI, quantum, and energy technologies are known and lab-made but face a scale-up bottleneck, not a discovery problem.
- Intel's high-k dielectric development at the 45nm node in 2007 highlighted that the breakthrough was scaling up processing, not discovering the material.
- The scale-up challenge has physical hurdles due to materials' complex, interconnected environments and informational hurdles from fragmented, siloed characterization data.
- AI and sensor-rich tools now enable real-time data processing, allowing systems like Atomscale to guide growth and use existing data more effectively.
- Atomscale uses physics-informed AI models to turn raw data into insights in real time, improving manufacturing reliability and knowledge accumulation across organizations.
- The future involves compressing scale-up timelines by leveraging interoperable materials data, moving from exploration to navigation and production.
China's ByteDance discovers new scaling law that could sustain AI boom
25 days ago
- ByteDance's Seed AI team found that AI agents can double their learning speed every three months through interaction with real-world environments.
- The AI industry is shifting from brute-force data and compute scaling to agentic AI, but post-deployment learning in real-world settings is poorly understood.
- To study this, ByteDance created EdgeBench, a benchmark with 134 long-term tasks requiring over 12 hours of continuous AI agent operation across various domains.

Hasty Briefsbeta

ai scaling

June gwern.net newsletter

February 2021 Gwern.net Newsletter

April 2021 newsletter

Overtraining as the path to human-like AI

Materials innovation has a scale-up problem, not discovery

China's ByteDance discovers new scaling law that could sustain AI boom