Language Models Pack Billions of Concepts into 12,000 Dimensions
6 hours ago
- #high-dimensional-geometry
- #language-models
- #machine-learning
- Language models like GPT-3 use a 12,288-dimensional embedding space to represent millions of concepts.
- The Johnson-Lindenstrauss lemma explains how high-dimensional spaces can preserve distances when projected into lower dimensions.
- Optimizing vector packing in high-dimensional spaces reveals practical limits and efficient configurations.
- High-dimensional spaces allow for quasi-orthogonal relationships, enabling nuanced semantic representations.
- Practical applications include efficient dimensionality reduction and embedding space design in machine learning.