Can LLMs Beat Classical Hyperparameter Optimization Algorithms?
6 hours ago
- #Machine Learning
- #LLM Agents
- #Hyperparameter Optimization
- Classical HPO algorithms (CMA-ES, TPE) outperform LLM-based agents in hyperparameter optimization on a fixed search space.
- LLM agents can edit training code directly but still lag behind classical methods even with advanced models like Claude Opus.
- Hybrid approach 'Centaur' combines CMA-ES state with LLM domain knowledge, achieving best results with smaller models (0.8B).
- LLMs complement classical optimizers effectively, rather than replacing them, based on search diversity and scaling analysis.