Study: AI responses to healthcare queries are nearly 76% accurate
12 hours ago
- #AI healthcare accuracy
- #LLM medical applications
- #patient safety concerns
- AI-powered chatbots respond to everyday health-related questions with about 76% accuracy, raising concerns about their trustworthiness in real-world applications.
- A study led by Penn State researchers explored AI accuracy in everyday medical queries, suggesting AI may work best for healthcare in trained physicians' hands rather than patients.
- The research involved a Diagnose-a-thon competition with 34 participants submitting 212 prompts and AI responses to health concerns, using models like ChatGPT-4o, Gemini-1.5 Pro, and Llama3-8b.
- Nine board-certified physicians evaluated AI responses for accuracy and harm potential, with prizes awarded for the most medically accurate and potentially harmful submissions.
- Findings will be presented at the 2026 ACM FAccT conference, highlighting the need for caution when using AI chatbots for health-related advice.