benchmarking

Bilingual

<antirez>
2 months ago
- Redis speed is a selling point, but comparative benchmarks are often misleading due to differences in data models, durability, and testing conditions.
- Fair comparisons require identical protocols and client libraries; otherwise, results reflect network stack and kernel rather than database performance.
- A benchmark comparing Redis and AerospikeDB on large EC2 instances (244 GB RAM) is unrealistic; a smaller m3.medium instance shows Redis achieving 32k ops/sec without pipelining.
- Using pipelining (e.g., 32 operations) boosts Redis performance dramatically: 250k ops/sec on a single core, outperforming the benchmark's multi-core result.
- Benchmarking simple GET/SET operations is insufficient; Redis excels at complex operations like ZADD for leaderboards and HyperLogLog for cardinality estimation, achieving 500k ops/sec combined.
- Better performance comparisons should focus on specific use cases, realistic instance types, and the number of instances needed to handle equivalent traffic.
Kimi K3, and what we can still learn from the pelican benchmark
2 months ago
- Moonshot AI released Kimi K3, a 2.8 trillion parameter model described as their most capable to date.
- K3 is currently available via website and API, with an open weight release promised by July 27, 2026.
- Self-reported benchmarks show K3 outperforms Claude Opus 4.8 max and GPT-5.5 high but loses to Claude Fable 5 and GPT-5.6 Sol.
- Artificial Analysis reports K3's Elo is 1547 on long-horizon knowledge work, with cost per task similar to GPT-5.6 Sol.
- Pricing is $3/million input tokens and $15/million output tokens, making it the most expensive Chinese AI lab model.
- The author uses a 'pelican riding a bicycle' SVG generation test to explore model characteristics.
- K3's test revealed high reasoning token usage (13,241 tokens), costing 25 cents, and a potential hidden system prompt.
- The pelican benchmark is limited, especially regarding agentic tool calling, and shouldn't be used to compare models.
- Despite limitations, the test serves as a forcing function to try models and reveals details like cost and SVG capability.
Kimi K3 is ranked 3rd on artificial analysis, only 2 points behind Sol
2 months ago
- Independent analysis of AI models and providers to help choose the best for specific use cases.
- Personalized model recommender for recommendations based on intelligence, speed, and cost priorities.
- Exploration of AI agents for general work, coding, customer support, and other tasks with comparisons across capabilities, pricing, and platform support.
- Premium plans offering expanded benchmark data, custom visualizations, and industry reports.
- Artificial Analysis Intelligence Index evaluating leading AI models, with sub-indexes for cost per task and frontier language model intelligence over time.
- Coding Agent Index measuring performance, cost, and execution time for leading coding agents on end-to-end software engineering tasks.
- Image, video, and speech leaderboards with 95% confidence intervals from arenas like Text to Image and Text to Speech.
- Agentic Index and Intelligence Evaluations covering agentic real-world work tasks, tool use, coding, reasoning, knowledge, and long-horizon tasks.
- Specific benchmarks: AA-Briefcase for long-horizon knowledge work, AA-Omniscience for knowledge and hallucination, GDPval-AA v2 for economically valuable tasks.
- Artificial Analysis Openness Index assessing model openness based on availability and transparency components.
- Output Tokens, Price and Cost, and Speed & Latency evaluations for leading AI models.
Benchmarking coding agents on Databricks' multi-million line codebase
3 months ago
- Databricks developed an internal coding benchmark using real engineering tasks from their codebase, evaluating AI coding agents on performance and cost.
- The benchmark identified three distinct capability tiers among models, indicating that high-cost models aren't always necessary for common tasks, with models like GLM 5.2 offering competitive quality at lower costs.
- Token costs often poorly predict overall task expenses due to variations in reasoning efficiency, with differences in harness performance highlighting the importance of context management.
- The team emphasized the need for custom benchmarks over public ones like SWE-Bench to accurately reflect internal needs and ensure optimizations don't hinder developers.
- Using merged pull requests as a foundation, tasks were rigorously filtered and specified to create a reliable benchmark, with manual validation to avoid issues like leaked solutions from git history.
- Findings support using cost-effective models for routine tasks and intelligent routing to balance efficiency and capability, with plans to expand the benchmark and automate model selection.
Claude Fable 5 Backlash Grows
3 months ago
- Anthropic re-released Claude Fable 5 on July 1 with stricter guardrails, leading to user backlash over reduced coding and debugging performance.
- Benchmark scores from BridgeMind show sharp declines in tasks like debugging (86.2 to 25.9) and refactoring, attributed to blocked tasks rather than weaker reasoning.
- Anthropic claims the underlying model is unchanged but admits tighter safety classifiers block more legitimate requests, routing them to Opus 4.8 instead.
- The model was initially launched in June, suspended briefly by regulators, and restored with usage limits and enhanced safety measures targeting cybersecurity tasks.
- Anthropic is collaborating with Amazon, Microsoft, and Google on a jailbreak severity framework, while international competition and user retention concerns grow.
An independent evaluation of TabFM, Google's tabular foundation model
3 months ago
- TabFM beats an Optuna-tuned XGBoost in a zero-shot setup on small-to-medium datasets.
- A multi-seed check led to demoting two narrow wins to ties due to measurement noise.
- The GPU memory footprint is about 16.95 GB, not 22.75 GB, after adjusting for XLA allocator artifacts.
- A bug causing crashes on multi-GPU systems was identified, fixed, and merged into the main repository.
- The CPU runs slower but handles more data, while GPU offers speed-ups of 14x to 42x.
- Fold-matched comparisons, noise measurement, and independent testing are critical for honest benchmarking.
Giving a domain a hill to climb: benchmarking as data activation
3 months ago
- Benchmarking is a form of data activation, transforming domain data into measurable tasks for models to be evaluated and trained on.
- Clear, verifiable metrics are essential for model improvement, as seen in coding and math, but complex domains like medicine lack inherent benchmarks.
- Activation starts with measurement: converting health data into scores for models can reveal knowledge gaps and drive improvement, even without model changes.
- Verifiers integrate benchmarking with reinforcement learning, turning scores into rewards and merging measurement with optimization, which amplifies both benefits and risks.
- The key challenge is making domains amenable to scale by converting messy data into verifiable tasks, with benchmarking as a conversion method.
- Different benchmark approaches exist on a spectrum: from expensive, raw-data-based methods (e.g., latchbio) to scalable, rubric-based ones (e.g., HealthBench) and lightweight multiple-choice setups (e.g., MedMarks).
- QuestBench highlights the importance of noticing missing information, which mirrors real-world tasks like medical diagnosis where identifying gaps is crucial.
- Curation remains a core challenge; expert judgment shapes benchmarks, but making it explicit in reward functions increases inspectability and transparency.
- Benchmark flaws become incentives when used as rewards, potentially leading models to learn distorted behaviors, emphasizing the need for careful design.
- Building benchmarks is a concrete form of data activation, involving creating verifiable tasks that enable scale to drive progress in complex domains.
Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
3 months ago
- Senior SWE-Bench treats agents like senior engineers by using realistic, natural language instructions instead of over-specified requirements.
- It introduces a validation agent that writes behavioral tests to evaluate tasks, adapting to submitted solutions.
- Bug tasks are based on tricky user reports requiring runtime investigation, such as debugging logs and reproduction steps.
- Scoring combines runtime correctness tests with quality metrics to assess tasteful code solutions.
- Tasks are sourced from PRs in diverse repositories and involve multi-phase, multi-stack features or bugs with significant runtime investigation.
- Instructions are naturally under-specified, with a median length 31% that of SWE-Bench Pro.
- Feature tasks can span multiple services, averaging 11 files touched per task, and are long-horizon, requiring hundreds of steps.
- A leaderboard shows top-performing models like Claude Opus 4.8 achieving a 24.0% solve rate, with frontier models failing over 75% of the time.

benchmarking

<antirez>

Kimi K3, and what we can still learn from the pelican benchmark

Kimi K3 is ranked 3rd on artificial analysis, only 2 points behind Sol

Benchmarking coding agents on Databricks' multi-million line codebase

Claude Fable 5 Backlash Grows

An independent evaluation of TabFM, Google's tabular foundation model

Giving a domain a hill to climb: benchmarking as data activation

Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers