Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers
a day ago
- #Reasoning Efficiency
- #Looped Transformers
- #Latent Chain-of-Thought
- Existing latent CoT methods underperform explicit CoT, especially with models larger than 1B parameters, and the performance gap increases with scale.
- Looped (recurrent-depth) Transformers reuse weights to increase computational depth without additional parameters, making them suitable for latent reasoning.
- LOTUS (Looped Transformers with parallel supervision on latents) is introduced as a method to bridge the gap between latent and explicit CoT.
- LOTUS uses a looped padded Transformer that processes K latent blocks in parallel for R iterations, with cross-entropy loss on each latent position's gold CoT-step token, similar to explicit CoT supervision.
- At the 3B scale, LOTUS is the first latent-CoT method to match explicit CoT performance while reducing thought-phase latency by 2.5x to 6.9x across tasks from math expressions to natural language.
- Projecting LOTUS's post-loop latents through the base language model head recovers gold reasoning steps and reveals alternative valid intermediate steps, indicating an interpretable and CoT-aligned latent space.
- Ablations confirm that both the looped backbone and parallel supervision on gold CoT tokens are essential for LOTUS's effectiveness.