Anthropic, Do Not A/B Test My Workflow
2 days ago
- #transparency
- #AI ethics
- #user experience
- Anthropic is conducting silent A/B tests on Claude Code, affecting user workflows without transparency.
- Users pay $200/month for Claude Code, expecting a stable, professional tool, not one subject to unannounced changes.
- The author discovered an A/B test named 'tengu_pewter_ledger' that alters plan mode functionality in Claude Code.
- Four variants exist: null, trim, cut, cap, with 'cap' being the most restrictive, limiting plans to 40 lines and removing context sections.
- Users are unknowingly enrolled in these tests, with no opt-in, notification, or toggle to control participation.
- The author was assigned the 'cap' variant, resulting in a degraded planning experience with no user input or discourse.
- Anthropic logs variant assignments and correlates them with plan length and task success, effectively using paying users as test subjects.
- This practice contradicts principles of transparency and responsible AI deployment, undermining user trust and control over their tools.