Science journalists find ChatGPT is bad at summarizing scientific papers
10 hours ago
- #science journalism
- #AI summarization
- #ChatGPT
- Science journalists and AAAS tested ChatGPT's ability to summarize scientific papers in a 'news brief' format similar to their 'SciPak' articles.
- ChatGPT could mimic the structure of SciPak briefs but often sacrificed accuracy for simplicity, requiring rigorous fact-checking.
- The study involved summarizing 64 complex papers from December 2023 to December 2024 using different prompts and GPT-4/GPT-4o models.
- AAAS concluded that while ChatGPT shows potential as a tool for science writers, it is not yet ready for full-scale use due to accuracy issues.
- Human biases may have influenced the evaluation, as journalists assessed a tool that could impact their core job functions.