OpenReviewer: A Specialized LLM for Generating Critical Scientific Paper Reviews
a day ago
- OpenReviewer is an open-source system for generating high-quality peer reviews of machine learning and AI conference papers.
- It uses Llama-OpenReviewer-8B, an 8B parameter model fine-tuned on 79,000 expert reviews from top conferences.
- The system extracts full text from PDFs (including equations and tables) and generates structured reviews following conference-specific guidelines.
- Evaluation on 400 test papers shows OpenReviewer produces more critical and realistic reviews compared to GPT-4 and Claude-3.5.
- OpenReviewer's recommendations closely match the distribution of human reviewer ratings, unlike other LLMs which tend to be overly positive.
- It provides rapid, constructive feedback to authors before submission, but is not intended to replace human peer review.
- OpenReviewer is available as an online demo and open-source tool.