Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

Authors: Alexi Gladstone, Heng Ji, Yilun Du

arXiv ID: 2607.27372

Problem: End-to-end training beat hand-designed stage decomposition in the deep learning revolution, but generative modeling never got there: scalable generative models all factor the generation procedure (for example, autoregressive and diffusion schedules) to handle multi-modal distributions, which prevents end-to-end generation and keeps inference step counts high.

Key Methodology:

Instead of factoring generation, factor the training loop: at each step, explore K candidate matches between model generations and data, and train on the best one, so predictions commit to modes rather than blurring them.
Validate in two settings: as a third pretraining axis (scaling exploration alongside parameters and data) on continuous and discrete domains (images, video, language), and as a standalone end-to-end generative modeling paradigm on control tasks.

Key Results:

Exploration scaling improves performance monotonically across domains, and gains increase with scale: from 7% to 36% as data scales and from 13% to 23% as models grow.
Efficiency gains: 4.1x FLOP efficiency, 6.2x sample efficiency, and 47% parameter efficiency, with efficiency gains more than doubling at 3x the compute.
Lifts the strongest image-generation recipe to 1.43 FID on ImageNet without guidance, near state of the art.
As a standalone paradigm, end-to-end reconstructive generative modeling matches diffusion on control tasks with 16-256x fewer inference steps.

Applied Context: For teams training or fine-tuning generative models, exploration budget becomes a tunable resource alongside data and parameters - a plausible lever for open-model labs closing capability gaps on smaller budgets. The end-to-end result also carries a serving-side implication: if generation can run end-to-end with 16-256x fewer steps, inference cost per artifact falls by a similar factor for the same quality on control-class tasks.