- GPT-5 failed to count the number of 'b's in 'blueberry', answering 3 instead of 2, sparking criticism on Bluesky.
- The failure is partly due to tokenization, where LLMs process text as tokens rather than individual letters, making letter counting difficult.
- Author conducted a systematic test of multiple LLMs, finding that most models correctly answered the 'strawberry' question (likely memorized), but GPT-5 Chat frequently failed on 'blueberry'.
- Other models like Claude Sonnet 4 and Opus 4.1 performed perfectly, while Gemini 2.5 Flash and open-source models showed inconsistent errors.
- The results suggest that GPT-5 Chat has a systemic issue beyond the router problem, and that LLMs can count letters but not reliably 100% of the time.