Bilingual

Simulating everything, sort of: The promise and limits of world models
13 days ago
- Large language models (LLMs) are no longer the sole focus of high expectations and investment in AI; world models are emerging as a significant new category.
- World models aim to enable AI systems to simulate the physical world or approximations of it, differing from LLMs that primarily handle language.
- Unlike LLMs that started with a chat interface and later sought use cases, world models are currently being developed with specific applications in robotics, research, and asset generation, though their final interfaces remain unclear.
- World models share architectural and improvement parallels with LLMs, but they are seen by some as addressing LLM limitations, such as the unrealistic expectation of achieving human-level intelligence through LLMs alone.
LeMario: Training a JEPA World Model on Super Mario Bros
12 days ago
- The author trained a Joint-Embedding Predictive Architecture (JEPA) model, named LeMario, on Super Mario Bros to learn world dynamics from pixels and actions.
- The model included a vision encoder, action encoder, and causal predictor with Adaptive LayerNorm Zero (AdaLN-Zero) for action conditioning, and used SIGReg to prevent representation collapse.
- LeMario outperformed baselines in short-horizon prediction on held-out episodes, showing it learned action-conditioned dynamics.
- However, when using reward-free planning with the Cross-Entropy Method (CEM) to reach distant image goals, Mario struggled with obstacles like jumps and navigation.
- Probing revealed the latent space encoded horizontal position well but vertical position weakly, explaining planning failures.
- Issues included latent distance not aligning with controllable progress, CEM exploiting model weaknesses, and differences from the original Push-T setup (e.g., scrolling camera, fewer episodes).
- Despite failures, the project provided insights into world models, emphasizing the importance of dataset, environment assumptions, and systematic testing.
Yann LeCun on AMI Labs, JEPA, and the AI World of 2030
10 days ago
- Yann LeCun founded AMI Labs in Paris to develop 'AI for the real world', focusing on predictive architectures like JEPA (Joint Embedding Predictive Architecture) for world models.
- LeCun believes current LLMs are insufficient for human-level intelligence and emphasizes world models that can predict physical consequences, enabling robots to handle new tasks without extensive training.
- He predicts AI systems with cat-level understanding of the physical world by 2030, driven by techniques like hierarchical JEPAs for planning and control.
- LeCun advocates for open-source AI to prevent power concentration, warning against restrictions that could stifle innovation and democracy.
- JEPA differs from generative AI by focusing on abstract representations to handle unpredictable real-world data, aiming for applications like predictive maintenance and optimal control.
- AMI Labs is working on hierarchical JEPA to enable hierarchical planning, with research being open-source while proprietary applications focus on industrial use.
- LeCun sees AI transforming education by shifting learning priorities, increasing demand for advanced skills, and enhancing roles like researchers and managers of AI agents.
- Healthcare applications, such as personalized treatment models and diagnostic AI, are anticipated, with Nabla as an early partner for administrative tasks.
- Intellectual inspiration comes from neuroscience, cognitive science, and physics, highlighting hierarchical abstractions as key to understanding intelligence.
- LeCun acknowledges cycles of AI hype and expects LLM expectations to renormalize within two years, followed by new waves based on world models.
Yann LeCun's Bet That Intelligence Starts in the World
12 hours ago
- The core debate in AI is whether intelligence begins with language or with building internal world models that predict consequences of actions.
- Yann LeCun's JEPA (Joint Embedding Predictive Architecture) predicts embeddings in latent space instead of raw pixels or tokens, avoiding blurry averages from next-frame prediction.
- Representation collapse, where models output the same embedding for all inputs, is a key challenge; approaches like Barlow Twins use cross-correlation matrices to avoid it.
- World models enable agents to predict future states, plan over action sequences, and act robustly in messy real-world environments, contrasting with behavior-cloning in VLA models.
- JEPA has shown promise in vision and video tasks (e.g., V-JEPA), but long-horizon planning remains difficult and likely requires hierarchical abstraction.
- The near-term future probably involves combining JEPA for world-state representation with LLMs for language interfaces, not replacing one with the other.

world models

Simulating everything, sort of: The promise and limits of world models

LeMario: Training a JEPA World Model on Super Mario Bros

Yann LeCun on AMI Labs, JEPA, and the AI World of 2030

Yann LeCun's Bet That Intelligence Starts in the World