RTK reports token savings, but our cost benchmarks disagree
4 days ago
- RTK compresses terminal output but does not guarantee cost savings; results vary from no savings to increased costs.
- Benchmarks on Terminal-Bench 2.1 showed minimal cost changes: Fable saved 5% while DeepSeek cost 5% more, with slight pass rate drops.
- The 'rtk gain' metric counts removed bytes, not actual token savings, and can misleadingly suggest cost reductions.
- RTK bugs (e.g., unsupported flags) can cause agent loops, significantly increasing turns and costs.
- Terminal output is a small fraction of total tokens; models already use efficient commands, limiting RTK's impact.
- Extra agent turns from RTK often outweigh compression savings, making it a niche optimization rather than a general cost-saving tool.