model comparison

Bilingual

13 Models and 4 Agents on SWE Tasks: Go, Java, Python, Rust, TS
2 months ago
- Fable 5 [high] model achieves the highest resolved rate at 64.5%, with a pass@5 of 78.4% and a cost of $4.40 per problem.
- GLM-5.2 [high] model has the highest pass@5 at 81.1%, while also maintaining a strong resolved rate of 62.9%.
- GPT-5.6 Sol [medium] offers a balance of high performance (62.3% resolved, 79.3% pass@5) with relatively low cost ($0.85) and token usage (605,340).
- MiMo V2.5 Pro model is the most cost-effective at $0.10 per problem, but has a lower resolved rate of 46.5%.
- MiniMax M3 model uses the most tokens per problem at 13,869,459, yet resolves only 47.2% of problems.
- Many models listed (from #18 onward) lack performance data, showing N/A for all metrics.
GPT-5.6 vs. Claude Fable 5 for Physical AI, which performs best?
2 months ago
- Physical AI requires correct physics modeling; models can compile and run while being physically wrong, and agentic AI worsens this by having agents write and pass their own tests that may rely on flawed simplifications.
- A study evaluated frontier AI models (OpenAI GPT 5.6 family and Anthropic Claude Fable 5) on five sealed modeling problems, including NASA's HL-20 flight vehicle, using a fixed Dyad AI harness to isolate model performance.
- Claude Fable 5 achieved the highest difficulty-weighted score (0.889), swept all core problems, and led on the HL-20, but cost $9.60 per trial, while GPT 5.6 Sol offered best value at $1.74 per trial with strong verification.
- The model's work-style fingerprints varied: Fable focused on verifying by attempting to falsify solutions, Sol emphasized precision but sometimes misread specs, Luna iterated heavily without external validation, and Terra economized but occasionally trimmed margins incorrectly.
- The harness provides more leverage than the model choice; swapping the harness improved scores by 0.366 points (from 0.533 to 0.899), more than double the 0.162 gap between best and worst models, emphasizing the importance of domain-specific tooling.
- The study concludes that for physical AI, selecting the right harness is the primary decision, followed by choosing the best affordable model within it, as the harness enforces proper verification and modeling practices.
"Drawing" the Mona Lisa with GPT-5.6, Claude, Gemini, and Grok
2 months ago
- Four frontier AI models (GPT-5.6 Sol, Claude Fable 5, Grok 4.5, Gemini 3.6 Flash) were tasked with drawing the Mona Lisa and other images using a colored-pencil toolset on a blank canvas.
- GPT-5.6 Sol was the overall best performer, producing the most detailed and appealing drawings, while Grok 4.5 performed poorly with mostly unusable output.
- Claude Fable 5 was the most expensive and slowest model, costing roughly 20x more than others yet not matching GPT-5.6's quality.
- Gemini 3.6 Flash scored highest on structural similarity (SSIM) for target reproductions but showed large score fluctuations and frequent editing.
- The experiment highlights the gap between frontier and open-weight models, reveals tool usage patterns, and demonstrates that more reviews do not guarantee better final results.
$100 AI Music Video: Claude Fable 5 vs. GPT-5.6 Sol
2 months ago
- An autonomous agent harness was built to let AI models create music videos with a given budget, song, and tools, comparing Claude Fable 5 and GPT-5.6 Sol at $25 and $100 budgets using 'Uptown Funk' as input.
- Both models successfully produced full-length videos, with Claude Fable 5 spending more on generation at $100 and using text-to-video only, while GPT-5.6 Sol employed varied approaches like image-to-video and mixing multiple video models.
- GPT-5.6 Sol had lower token costs ($3-4) compared to Claude Fable 5 ($17-25), leading to lower total costs, and showed more inventive editing techniques, though all videos struggled with consistency, tempo matching, and lacked iterative editing.
Price per 1M tokens is meaningless
3 months ago
- Comparing AI model costs by price per 1M tokens is meaningless due to differences in tokenizers and token efficiency.
- Each frontier lab uses its own tokenizer, causing the same text to be split into varying token counts, making direct price comparisons unreliable.
- Token efficiency—how much is achieved per token—varies widely, especially with 'thinking' tokens in chain-of-thought processes, significantly impacting overall costs.
- A benchmark table shows models like GPT-5.5 can have lower cost per task despite higher token prices, while cheaper per-token models like GLM-5.2 may be less token-efficient.
- DeepSeek V4 Pro stands out for extremely low cost per task despite lower intelligence scores, highlighting cost-efficiency disparities.

model comparison

13 Models and 4 Agents on SWE Tasks: Go, Java, Python, Rust, TS

GPT-5.6 vs. Claude Fable 5 for Physical AI, which performs best?

"Drawing" the Mona Lisa with GPT-5.6, Claude, Gemini, and Grok

$100 AI Music Video: Claude Fable 5 vs. GPT-5.6 Sol

Price per 1M tokens is meaningless