Compaction can save 50% due to cache-read cost
11 hours ago
- #Agent Traces Compaction
- #AI Cost Optimization
- #Long-Context Agents
- AI costs are rising, leading companies to replace token leaderboards with daily quotas and seek cost reduction strategies.
- Compacting agent traces after periods of user inactivity can save up to 50% on coding workloads, using a model that runs at over 50k tok/s without harming user experience.
- In long-running conversations, input tokens (not output tokens) dominate costs due to cache reads and rewrites, with costs growing quadratically with session length.
- Compaction on cache miss reduces cache-read costs from quadratic to linear and minimizes the expense of rewriting large caches.
- Traditional compaction methods are too slow (1-2 minutes), but Relace Compact operates at over 50k tok/s, making it fast and imperceptible for users.
- API token pricing includes cache writes, cache reads, and output tokens, with cache reads becoming dominant in long-context settings.
- Cache rewrites occur when cached data expires (TTL), adding significant costs, especially in idle sessions.
- Compaction is most effective during cache misses, as it reduces the size of expensive cache writes and avoids additional latency.
- Relace Compact is a fast, specialized model for token-level classification, integrated into Jacq and available via API, offering estimated savings of over 56%.
- Conclusion: Compaction on cache miss addresses quadratic cost growth from input tokens, with Relace Compact enabling efficient, latency-free implementation for major cost reductions.