Fast and Accurate Long Text Generation with Few-Step Diffusion Language Models
a day ago
- #Natural Language Generation
- #Machine Learning
- #Diffusion Models
- FS-DFM (Few-Step Discrete Flow-Matching) is introduced for fast and accurate long text generation.
- Autoregressive language models (ARMs) are limited by serial token generation, affecting throughput and latency.
- Diffusion Language Models (DLMs) parallelize across positions but require many steps for high-quality output.
- FS-DFM reduces the number of sampling steps needed while maintaining quality, achieving up to 128 times faster sampling.
- The model is trained to be consistent across step budgets, allowing fewer steps without quality loss.
- Strong teacher guidance and a reliable update rule ensure stable and accurate few-step sampling.
- FS-DFM achieves perplexity parity with traditional methods using significantly fewer steps.