Explorative Modeling – A new axis for pre-training
5 hours ago
- Deep learning revolutionized AI with end-to-end training, but generative models still avoid this approach due to the difficulty of handling multi-modal distributions.
- Explorative Modeling (XM) factors the training loop by exploring K candidate matches between model generations and data, training on the best to commit to modes instead of blurring them.
- Increasing exploration adds a third pretraining axis beyond parameters and data, improving performance across images, video, and language as scale increases.
- Gains from exploration grow with scale: from 7% to 36% with data scaling and from 13% to 23% with model scaling; efficiency improves up to 4.1x in FLOP, 6.2x in samples, and 47% in parameters.
- XM achieves a near-state-of-the-art 1.43 FID on ImageNet without guidance and enables end-to-end reconstructive modeling, matching diffusion on control tasks with 16-256x fewer inference steps.