2 months ago
- AI cost in tokens and dollars is a major concern, with examples like Uber exceeding its budget quickly.
- Coding agents involve different token types: input, cached input, output, and reasoning tokens, each with different prices and implications.
- Cached input tokens are the cheapest but can dominate costs due to repeated use in agentic loops, while output tokens are the most expensive.
- Providers like Anthropic and OpenAI handle caching differently, affecting billing and cache optimization strategies.
- Token usage patterns vary by application; coding agents typically have high cached token volumes, unlike chatbots or RAG systems.
- AI subscriptions lack transparent token allowances, making telemetry crucial for monitoring and managing token consumption.
- Investing in good telemetry is essential for understanding costs, especially when switching between multiple AI providers.
- AI budgets depend on token volume, not just pricing; cached tokens can become a major cost due to high usage volume.
- Coding agents significantly influence token volume; choosing and configuring them carefully is as important as selecting an LLM provider.
- Forecasting future token usage requires telemetry and will be explored in a follow-up article to avoid budget overruns.