Does Speaking to Agents Like Cavemen Save 65% of Tokens? We Test
2 days ago
- JetBrains tested the Caveman skill on Claude Code using SkillsBench to verify token-saving claims.
- Claimed 65% output token savings only apply to chat-style prose, not agentic tasks.
- Actual token savings on multi-step agent work was about 8.5%, not 65%.
- No detectable quality degradation: 82 paired tasks showed statistically indistinguishable results.
- Cost savings were about 10% in expectation but fragile due to outlier trials.
- Recommendation: use Caveman for fun without quality loss, but don't expect large savings on daily agent tasks.