Why are cached input tokens cheaper with AI services?
2 days ago
- Cached input tokens are cheaper because the GPU avoids recomputing the same intermediate state (KV cache) for repeated conversation prefixes.
- Key-value caching (KV caching) stores the model's internal state after processing initial messages, so subsequent calls with the same prefix skip redundant computation.
- Providers pass these compute savings to users as lower per-token prices for cache hits compared to cache misses.
- To maximize cache hits, avoid changing inference settings or previous messages between prompts in an application.