How to win a beer with high-dimensional statistics
2 days ago
- A paper by Dhruva Karkada on data statistics went viral for showing that LLM embeddings of the months form a circle and a circulant Gram matrix; word2vec approximates this and an analytical theory matches it.
- The author bet a beer that they could find other seemingly unrelated words that also form a circle and circulant Gram matrix using word2vec embeddings.
- They searched a vocabulary of 25,000 words for a set of 10 words, using an iterative search that dropped the worst point and selected replacements to match a target circulant Gram matrix.
- The search succeeded, producing a clear circular structure from random-looking words, demonstrating that spurious geometric patterns can be found without special semantic relationships.
- Caveats: the spurious circle had smaller off-diagonal amplitudes than the months, and the method likely would not work for larger sets like 50 words, so some findings such as years from 1700-2020 remain robust.
- The result warns that pursuit-style searches for low-dimensional PCA geometries can produce misleading patterns unless statistical constraints are strong enough to avoid chance matches, impacting automatic feature-finding and scalable interpretability research.