ai benchmarks

Bilingual

The famous o3 "GeoGuessr" prompt did not work
2 months ago
- The elaborate 'GeoGuessr' prompt for OpenAI's o3 model did not improve geolocation performance compared to a simple default prompt.
- A benchmark of 200 images showed the default prompt performed slightly better in median and mean distance from actual location.
- The author warns that prompt engineering can create a false sense of improvement, as models often rationalize their own mistakes.
- The benchmark was easy and cheap to run ($15, six hours), yet no one had tested the prompt's effectiveness during the initial hype.
- o3's geolocation abilities have not transferred to newer models like GPT-5.4 and GPT-5.5, which performed worse in the same benchmark.
- The author addressed concerns about images being in training data, noting that public domain images still allow useful comparisons.
Can modern LLMs actually count the number of b's in "blueberry"?
2 months ago
- GPT-5 failed to count the number of 'b's in 'blueberry', answering 3 instead of 2, sparking criticism on Bluesky.
- The failure is partly due to tokenization, where LLMs process text as tokens rather than individual letters, making letter counting difficult.
- Author conducted a systematic test of multiple LLMs, finding that most models correctly answered the 'strawberry' question (likely memorized), but GPT-5 Chat frequently failed on 'blueberry'.
- Other models like Claude Sonnet 4 and Opus 4.1 performed perfectly, while Gemini 2.5 Flash and open-source models showed inconsistent errors.
- The results suggest that GPT-5 Chat has a systemic issue beyond the router problem, and that LLMs can count letters but not reliably 100% of the time.
Ollama vs. Llama.cpp – Quick Benchmark
3 months ago
- Ollama and llama-server benchmark on Tesla V100 using Qwen3 8B Q4_K_M GGUF.
- Generation speeds nearly identical: llama-server at 111.00 tok/s, Ollama at 110.61 tok/s.
- Prompt processing faster in llama-server: 1255.29 tok/s vs 1028.02 tok/s for Ollama.
- Ollama used prompt caching, inflating results if not disabled via unloading.
- Benchmark limited: short prompt (60 tokens), one model/GPU, no concurrent requests.
- llama-server recommended for long prompts; Ollama comparable for generation speed.
- vLLM suggested as another candidate but unsupported on V100's Volta architecture.

ai benchmarks

The famous o3 "GeoGuessr" prompt did not work

Can modern LLMs actually count the number of b's in "blueberry"?

Ollama vs. Llama.cpp – Quick Benchmark