3 months ago
- Coding agents send similar large prompts repeatedly, making prompt caching crucial for efficiency and cost.
- The KV cache stores key and value tensors per token and layer, enabling reuse of processed prefixes.
- Session affinity keeps the cache on the same GPU; distributed caching spreads it across workers, offering flexibility but with system overhead.
- Tree-structured sessions (e.g., branches and rewinds) cause only partial prefix overlap, reducing cache efficiency.
- Explicit caching (e.g., Anthropic's cache_control) requires client marks; automatic prefix caching finds reusable prefixes by itself.
- Changes to tool definitions, order, or loadouts invalidate the cached prefix, often wasting tens of thousands of conversation tokens.
- Short cache TTLs (e.g., Anthropic's 5-minute default) cause misses during long activities like builds or meetings.
- Cache misses increase costs significantly: a 100,000-token history is re-processed at full input price, plus potential write fees.
- Pi avoids aggressive pruning to keep the cached prefix stable; compaction is done only when needed, but it resets the cache.
- Pi monitors cache performance with indicators like hit rate, re-billed tokens, and miss warnings to help users manage costs.