Measuring the Impact of AI on Experienced Open-Source Developer Productivity
10 months ago
- #Randomized Controlled Trial
- #Software Development
- #AI Productivity
- Randomized controlled trial (RCT) shows AI tools slow down experienced open-source developers by 19%.
- Study aims to understand AI's real-world impact on software development, contrasting with benchmarks.
- 16 experienced developers worked on 246 real issues, with or without AI tools, recording their screen time.
- Developers expected AI to speed them up by 24%, but even post-study, they believed AI sped them up by 20%.
- Five factors identified as likely contributors to the slowdown, with many experimental artifacts ruled out.
- Discussion highlights discrepancies between benchmark scores, anecdotal reports, and RCT results.
- Framework proposed to reconcile different sources of evidence on AI capabilities.
- Future studies planned to track AI's impact on developer productivity, signaling broader AI R&D acceleration risks.