The Annotated JEPA
4 hours ago
- #Self-Supervised Learning
- #Representation Learning
- #Computer Vision
- JEPA (Joint Embedding Predictive Architecture) is Yann LeCun's self-supervised learning approach for training models to understand the world without labels by predicting in representation space, avoiding trivial solutions and irrelevant details.
- I-JEPA, an image instantiation of JEPA, trains a context encoder to predict representations of masked target regions using a target encoder updated via exponential moving average (EMA) to prevent collapse, with a predictor minimizing MSE in embedding space.
- Key design choices in I-JEPA include patch tokenization, masking strategies with context and target blocks without overlap, and applying masking in the target encoder output to ensure high-level semantic representations.
- V-JEPA extends JEPA to video by masking spatiotemporal 3D blocks, learning representations that capture motion and content without pixel reconstruction, enabling strong performance in video understanding tasks.
- V-JEPA 2 further demonstrates that JEPA can support understanding, prediction, and planning by using the learned representations for classification, future latent state forecasting, and action-conditioned planning in robotics.
- LeJEPA introduces a theoretically grounded variant using Sketched Isotropic Gaussian Regularization (SIGReg) to enforce an isotropic Gaussian embedding distribution, aiming to eliminate heuristics like EMA teachers and provide stable training.
- JEPA is framed as an energy-based model, where prediction error in representation space defines energy, and it serves as a core component for world models that enable planning by searching over abstract latent states rather than pixels.
- The architecture contrasts with autoregressive language models by focusing on raw sensory data (e.g., images, video) to learn physical world structures, positioning JEPA as a foundation for autonomous intelligence systems involving prediction and action.