- Token counts for the same content vary significantly between AI models due to different tokenizers, which is not reflected in the advertised price per million tokens.
- Anthropic's new tokenizer (used in Claude models like Opus 4.8 and Sonnet 5) increases token counts by about 30-32% compared to its old tokenizer, despite unchanged headline prices, leading to higher effective costs.
- The tokenizer divergence is most pronounced for code, especially TypeScript, where Claude's new tokenizer uses 1.73x more tokens than OpenAI's GPT (with o200k tokenizer), compared to 1.4x for English prose.
- Effective pricing should account for tokenizer differences; for example, Claude Opus 4.8 has an effective price of $7.50/$37.50 per million tokens for input/output, compared to its headline $5.00/$25.00, when adjusted for token count.
- Real-world measurements show GPT's o200k tokenizer is the most efficient for English and code, while Gemini 3 Flash remains cost-effective despite a slightly heavier tokenizer due to low headline prices.
- To accurately compare model costs, users should measure token counts on their specific content, treat tokenizer changes as hidden price hikes, and evaluate costs per task rather than per token.