We Tried to Detect Bots in Comments. We Found a More Interesting Problem

20 hours ago

The attempt to differentiate bot from human comments revealed that AI tells are often structural and quickly absorbed into human writing or AI training, making reliable detection difficult.
The study found that comment quality, not authorship, is the more useful metric, with 'bot-like' traits often correlating with low effort and little added value to discussions.
Comments were categorized into three groups: those adding value (specific and engaging), noise (low-effort reactions), and a large middle ground (coherent but generic, hard to classify as bot or human).
The post's content significantly influences comment quality; data-heavy posts tend to elicit more specific responses, while opinion posts encourage generic replies, affecting perceived 'botness.'
As content generation becomes cheaper, the bottleneck shifts from contribution to curation, requiring platforms to focus on trust, attention protection, and value-based ranking to sustain quality.
A proposed framework for platforms emphasizes content observability, contribution quality, behavioral credibility, contextual calibration, and adaptive feedback loops to prioritize readers over engagement metrics.
Bot detection remains relevant for spam and impersonation, but for the bulk of generic content, platforms should shift focus from authorship to whether content adds value for users.

Hasty Briefsbeta