LeJEPA
5 days ago
- #Self-Supervised Learning
- #AI Research
- #Machine Learning
- LeJEPA introduces a theoretically grounded self-supervised learning method without heuristics.
- Identifies isotropic Gaussian as the optimal distribution for embeddings to minimize prediction risk.
- Introduces Sketched Isotropic Gaussian Regularization (SIGReg) to constrain embeddings.
- Combines JEPA predictive loss with SIGReg for a simplified, scalable, and stable training objective.
- Achieves 79% accuracy on ImageNet-1k with a ViT-H/14 model in linear evaluation.
- Requires only ~50 lines of code for a distributed training-friendly implementation.
- Validated across 10+ datasets and 60+ architectures, showing broad applicability.