HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88
7 hours ago
- #LLM
- #ATS
- #Recruiting
- HackerRank's open-source ATS tool shows significant score inconsistencies when evaluating the same resume, with scores ranging from 66 to 99.
- The tool's scoring includes categories like open source contributions, projects, work experience, and technical skills, with a heavy weight on open source and personal projects.
- Technical skill scores are consistent due to a checklist approach, while project and experience scores vary widely, revealing LLM limitations in subjective judgment.
- Work experience scores are consistently maxed at 25/25 for all levels, from interns to senior engineers, due to a vague rubric, making them meaningless.
- The non-deterministic scoring, even at low temperature settings, can cause qualified candidates to fail screening due to luck, not merit.
- AI screening tools that rely on LLMs for nuanced evaluations may filter based on randomness rather than quality, disadvantaging candidates without public code.