Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
4 hours ago
- Scaling laws state that larger language models with more data are more capable, and Mixture-of-Experts (MoE) architectures activate only a fraction of a large stored parameter bank per token.
- Deployed models encounter live data not in their training set; conventional models cannot learn from it due to frozen weights and instead rely on prompt-based retrieval or instruction.
- The proposed Infinite-Parameter LLM uses a compact hypernetwork to generate low-rank modulations of a shared base network from live data, enabling weight generation rather than fixed storage.
- A Bayesian belief over the hypernetwork's latent code is updated online, allowing effective weights to evolve during a session rather than being fixed after one read.
- Storing knowledge in weights rather than the prompt amortizes compute, frees the context window, persists across turns, and generalizes better than in-context learning.
- An evaluation protocol is specified to test the model against in-context learning and retrieval methods.