Pre-Training Large Memory Language Models with Internal and External Knowledge
9 months ago
- #Language Models
- #Knowledge Representation
- #Machine Learning
- Proposes Large Memory Language Models (LMLM) that store knowledge in both internal weights and an external database.
- Uses strategic masking of externally retrieved facts to encourage targeted lookups over memorization.
- Achieves competitive performance with larger models while offering editable and verifiable knowledge bases.
- Represents a shift in how language models manage factual knowledge.