self-supervised learning

The Annotated JEPA

3 months ago

JEPA (Joint Embedding Predictive Architecture) is Yann LeCun's self-supervised learning approach for training models to understand the world without labels by predicting in representation space, avoiding trivial solutions and irrelevant details.
I-JEPA, an image instantiation of JEPA, trains a context encoder to predict representations of masked target regions using a target encoder updated via exponential moving average (EMA) to prevent collapse, with a predictor minimizing MSE in embedding space.
Key design choices in I-JEPA include patch tokenization, masking strategies with context and target blocks without overlap, and applying masking in the target encoder output to ensure high-level semantic representations.
V-JEPA extends JEPA to video by masking spatiotemporal 3D blocks, learning representations that capture motion and content without pixel reconstruction, enabling strong performance in video understanding tasks.
V-JEPA 2 further demonstrates that JEPA can support understanding, prediction, and planning by using the learned representations for classification, future latent state forecasting, and action-conditioned planning in robotics.
LeJEPA introduces a theoretically grounded variant using Sketched Isotropic Gaussian Regularization (SIGReg) to enforce an isotropic Gaussian embedding distribution, aiming to eliminate heuristics like EMA teachers and provide stable training.
JEPA is framed as an energy-based model, where prediction error in representation space defines energy, and it serves as a core component for world models that enable planning by searching over abstract latent states rather than pixels.
The architecture contrasts with autoregressive language models by focusing on raw sensory data (e.g., images, video) to learn physical world structures, positioning JEPA as a foundation for autonomous intelligence systems involving prediction and action.

Hasty Briefsbeta

self-supervised learning

Inertia-1: An Open Exploration to a Unified Motion Foundation Model

The Annotated JEPA