Do LLMs identify fonts?
9 months ago
- #benchmark
- #font-identification
- #LLM
- dafont.com is a comprehensive font collection website with a forum for font identification requests.
- A benchmark was created to test if LLMs (gpt-4o-mini and gemini-2.5-flash-preview-05-20) could identify fonts from images not yet identified by the community.
- The benchmark avoids contamination by only using unidentified fonts and comparing LLM predictions to community answers later.
- LLMs were provided with the image, thread title, and description to focus on the correct font in complex cases.
- Performance was measured using top-$k$ accuracy, allowing up to five guesses per font.
- Results showed poor performance, raising questions about task difficulty or evaluation fairness.
- The implementation involved Python scripts for scraping and prompting, GitHub Actions for automation, and Observable Framework for the dashboard.
- The live benchmark is hosted on GitHub Pages.