Learning to solve hard problems in RL for LLMs by never giving up
11 hours ago
- The Matthew Effect describes how RL improves performance on tasks proportionally to initial competence, favoring easy problems over hard ones.
- The cause is signal efficiency: too much compute is spent on easy problems due to rare incorrect completions, not just undersampling hard ones.
- Never Give Up (NGU) adapts sampling by using small k for easy problems and resampling hard problems until solved, reallocating compute.
- NGU outperforms standard GRPO on math and code benchmarks, especially on hardest subsets, without harming easy performance.
- NGU is robust to plasticity issues; models can recover from early bad samples.
- Limitations include potential off-policy staleness and reduced effectiveness on mostly-hard problem distributions.