How big are our embeddings now and why?
8 days ago
- #embeddings
- #machine-learning
- #AI
- Embeddings are compressed numerical representations of features like text, images, and audio, used for machine learning tasks such as search, recommendations, and classification.
- Initially, embeddings were around 200-300 dimensions, but with advancements like BERT, sizes increased to 768 dimensions due to GPU optimizations and the need for parallel computation across attention heads.
- The rise of HuggingFace and API-based models like OpenAI's (1536 dimensions) has standardized and commoditized embeddings, making them more accessible.
- Current embedding sizes range from 768 to 4096 dimensions, influenced by model architectures and the need for efficient storage and retrieval.
- Techniques like matryoshka representation learning and research on embedding truncation are emerging to optimize the trade-off between model size and performance.