- HackerRank released an open-source ATS (Applicant Tracking System), which uses an LLM to evaluate resumes with scores out of 100.
- Testing the ATS with the same resume produced wildly inconsistent scores (e.g., from 66 to 99), making hiring outcomes heavily dependent on luck rather than merit.
- The scoring system awards points for categories like open source contributions, personal projects, work experience, and technical skills, but project and experience evaluations lack consistency and clear rubrics.
- LLM-based evaluations struggle with subjective judgments, such as project quality, leading to high variability, while objective criteria like technical skills remain consistent.
- The author warns that AI screening tools may filter candidates based on random variance rather than true quality, potentially disadvantaging experienced engineers without extensive open-source or project visibility.