benchmarks

Bilingual

AI Has Passed Every Exam. It Has Never Had an Idea
6 hours ago
- AI excels at passing exams and finding answers, achieving superhuman results on tests like Humanity's Last Exam and the International Math Olympiad.
- AI contributes to scientific discoveries (e.g., AlphaFold, GNoME) but only when humans decide which questions to ask; it has not independently chosen important questions.
- The contradiction between AI's superb answer-finding and lack of question-posing is resolved by recognizing that all benchmarks measure only answer-finding, not question-choosing.
- Historical breakthroughs like Einstein's theory of relativity came from questioning existing assumptions, not just better answer-finding.
more...
Measured LLM inference speeds on Apple Silicon, with raw data (CC BY 4.0)
5 days ago
- Benchmarks for LLM inference on Apple Silicon were measured using a production fleet with Ollama API, fixed prompt, and 512-token generation budget.
- Results for M4 Mac mini (16 GB, 120 GB/s) include models from 3B to 14B parameters, with generation speeds ranging from 46.7 tok/s (Llama 3.2 3B) to 11.7 tok/s (Qwen 2.5 14B).
- Future benchmarks will cover M4 Pro, M4 Max, and M3 Ultra chips, with performance expected to scale with memory bandwidth.
- Methodology: Ollama 0.31.2 on macOS 15.3.1, default Q4_K_M quantization, 3 recorded runs after warm-up, median speeds reported. Raw data available under CC BY 4.0.
Claude Opus 5: The System Card
7 days ago
- Claude Opus 5 aims to match or exceed Fable 5's performance on many tasks while being faster and half the price.
- Opus 5 shows substantial gains over Opus 4.8 in agentic coding, computer use, and long-horizon knowledge work, setting new state-of-the-art benchmarks.
- Opus 5 lacks full 'Juice' for cyber offense and bio threats due to deliberate avoidance of cyber training and smaller model size.
- Safety classifiers trigger 85% less often than Fable's, permitting source code vulnerability analysis but blocking binary vulnerability discovery.
more...

benchmarks

AI Has Passed Every Exam. It Has Never Had an Idea

Measured LLM inference speeds on Apple Silicon, with raw data (CC BY 4.0)

Claude Opus 5: The System Card