Show HN: PILF, The ultimate solution to catastrophic oblivion on AI models
10 months ago
- #adaptive learning
- #hyperparameter optimization
- #machine learning
- PILF is a cognitive learning framework that transforms fixed hyperparameters into dynamic policies based on data 'surprise'.
- It dynamically adjusts learning rate and model capacity in real-time, replacing static hyperparameters with data-driven policies.
- PILR-S focuses on dynamically adjusting the learning rate based on 'surprise', using a Gaussian function for modulation.
- PILF extends this to MoE architectures, dynamically deciding both the number of experts to activate and the learning rate.
- The framework aims to unify learning, ignoring, and rejecting mechanisms, improving efficiency and mitigating catastrophic forgetting.
- Experiments are conducted using lightweight Vision Transformers, comparing different variants on datasets like CIFAR-10 and MNIST.
- The project is open-source, licensed under AGPLv3, and requires PyTorch and the sigma-pi package for core calculations.