8 hours ago
- A student used cosine similarity on top word frequencies to detect similar or fake Hacker News accounts.
- The author replicated this method using Redis Vector Sets and the Burrows-Delta technique.
- Data preprocessing involved downloading HN comments, converting Parquet files to JSONL with user word frequency tables.
- The Burrows-Delta method computes relative frequency, then z-scores using global mean and standard deviation for each word.
- The optimal number of top words is between 150 and 500; using too many captures content rather than writing style.
- Validation showed that splitting a user's comments into two sets and comparing them correctly identifies the same user.
- Visualization of vectors reveals over- and under-used words, highlighting differences between native and non-native speakers.
- The code and a demo site are available, with filtering by word count to help detect fake or throwaway accounts.