Eggroll: Novel general-purpose machine learning algorithm provides 100x speed
a day ago
- #large language models
- #evolution strategies
- #machine learning
- EGGROLL is a novel evolution strategies (ES) algorithm designed for large-scale neural network training.
- It uses low-rank matrix perturbations to reduce computational and memory costs significantly.
- EGGROLL achieves a hundredfold increase in training speed for billion-parameter models compared to naïve ES.
- The algorithm enables efficient training of models with pure integer datatypes and nonlinear RNNs without activation functions.
- EGGROLL is competitive with GRPO for improving LLM reasoning and supports stable pre-training of novel architectures.
- The method is general-purpose, allowing optimization of any inference system with a defined fitness function.
- Experimental results show EGGROLL's effectiveness in pretraining and fine-tuning tasks, including GSM8K and countdown tasks.
- Future work includes optimizing neurosymbolic systems and multi-agent LLMs with EGGROLL.