Recursive Language Models (RLMs)
a day ago
- #Context Rot
- #Recursive Language Models
- #Long-context Processing
- Recursive Language Models (RLMs) proposed as an inference strategy to handle unbounded input context length by decomposing and recursively interacting with input context.
- RLMs allow language models to recursively call themselves or other LLMs before providing a final answer, mitigating 'context rot'.
- RLMs outperform GPT-5 on difficult long-context benchmarks, demonstrating significant performance improvements.
- Context rot is described as a degradation in model performance as context length increases, a phenomenon not fully captured by current benchmarks.
- RLMs use a Python REPL environment to interact with context, enabling programmatic processing and recursive sub-queries.
- RLMs show promise in handling extremely long contexts (up to 10M+ tokens) without requiring model architecture changes.
- Key benefits include improved performance on semantic and counting tasks, and the ability to scale with context length.
- RLMs differ from agents by allowing the model to decide how to decompose the problem, rather than relying on human intuition.
- Future work includes optimizing RLM implementations for speed and efficiency, and exploring training models explicitly for recursive reasoning.