Bilingual

Kimi K3's Design Secret May Be in Its Thinking Traces
3 days ago
- Kimi K3 ranks 1st on single-shot Frontend Arena with 1392 Elo, a significant jump over previous Kimi models.
- It uses over 12x more thinking tokens than Claude Opus 4.8 and double that of Kimi K2.6.
- Kimi K3's performance is attributed to a unique chain-of-thought that simulates an AI agent, iterating on designs like an agent would.
- It writes over 10x more code during reasoning than other Kimi models, effectively 'thinking in code'.
- The model leverages a strong learned index to validate Unsplash image IDs during reasoning, leading to better image usage.
- This strategy trades tokens for intelligence, resulting in slower but higher-quality outputs on a preference-speed Pareto frontier.
ARC-AGI Leaderboard
3 days ago
- ARC-AGI-3 measures AI agents' ability to adapt on the fly to novel interactive environments, evolving from earlier passive fluid intelligence tests.
- The leaderboard scatter plot shows the relationship between cost-per-task and performance, emphasizing efficiency as a key measure of intelligence.
- Three types of data are plotted: Reasoning Systems Trend Lines (showing effect of reasoning time), Base LLMs (single-shot inference), and Kaggle Systems (competition-grade submissions under strict cost constraints).
- Only systems costing less than $10,000 to run are shown; incomplete test outputs are marked incorrect, and preview results are unofficial.
What's new in Claude Opus 5
4 days ago
- Claude Opus 5 is a significant improvement over Opus 4.8, with gains in deep reasoning, agentic tasks, long-horizon tasks, and test-time compute scaling.
- Thinking is enabled by default on Claude Opus 5; disabling thinking is restricted to effort levels high or below, and attempting to disable it with xhigh or max effort returns a 400 error.
- The model supports a 1M token context window, 128k max output tokens, and an effort ladder from low to max for controlling reasoning depth.
- Mid-conversation tool changes are in beta, allowing tools to be added or removed between turns while preserving the prompt cache.
- A new 'default' mode for fallbacks applies Anthropic's recommended fallback models by refusal category, requiring a beta header.
- The minimum cacheable prompt length is reduced from 1,024 to 512 tokens, enabling caching for shorter prompts without code changes.
- Fast mode is in research preview for Claude Opus 5 on the API only, priced at $10 per million input tokens and $50 per million output tokens.
- In agentic sessions, the model narrates progress more often, delegates to subagents more readily, and verifies its own work, so verification instructions from earlier models cause over-verification.
- Claude Opus 5 delivers frontier intelligence at half the cost of Claude Fable 5, with pricing unchanged from Opus 4.8 at $5 per million input tokens and $25 per million output tokens.
- The model is available on Anthropic API, Amazon Bedrock, Google Cloud Vertex AI, and Microsoft Foundry; migration requires updating the model ID and reviewing behavior changes.
Schema Harness Achieves ~99% on Arc‑AGI‑3 Public
12 days ago
- Schema harness enables frontier models to achieve ~99% performance on ARC-AGI-3 Public by treating latent world representations as executable programs, allowing interpretable, verifiable, and searchable modeling.
- ARC-AGI-3 challenges agents to infer game rules, objects, and goals from raw 64x64 color grids without explicit instructions, using the Relative Human Action Efficiency (RHAE) metric for evaluation.
- The approach combines state grounding (identifying objects and variables from observations) and mechanism discovery (learning transition rules), solved jointly within an editable program that mirrors physicist-like reasoning.
- Controlled comparisons show Schema improves Claude Code baseline by 56.15%, with efficiency gains from verifying models against complete history and planning inside reusable simulators, reducing action counts significantly.
- Case studies highlight Fable models discovering world models more efficiently than Opus by better experimental decisions, representational revisions, and earlier identification of key mechanisms.
- The harness reduces theory usage costs through persistence and searchability, while the underlying model affects discovery efficiency; search completeness depends on the accuracy of the world model.
- Self-reported results on the Public set (98.98% with Claude, 95.35% with Sol) indicate progress but lack independent verification; extrapolation to Semi-private performance remains uncertain.
AI bots ignore evidence. Can we trust them with science?
24 days ago
- AI chatbots like ChatGPT, Gemini, and Grok failed to update predictions based on experimental evidence in a pen demonstration, sticking to incorrect assumptions.
- A study showed AI agents ignored evidence in 68% of scientific reasoning tasks, made unsupported claims in 53%, and used contradictory evidence to change output only 26% of the time.
- AI systems lack an iterative reasoning process similar to human scientists, often refusing to revise hypotheses despite clear evidence, limiting their reliability in science and medicine.
- Researchers developed a benchmark to evaluate AI agents' reasoning process rather than just outcomes, revealing gaps in their ability to incorporate new data transparently.
- Reasoning models, which mimic step-by-step thinking, may not truly reason but imitate patterns without verification, making it hard to trust their problem-solving process.
- AI is best suited for well-defined tasks in science, not open-ended reasoning, contradicting claims of emergent intelligence and raising concerns about knowledge erosion.
- Understanding AI's limitations allows for improvement towards meaningful discoveries, though current systems risk undermining scientific trust and transparency.

Hasty Briefsbeta

ai reasoning

Kimi K3's Design Secret May Be in Its Thinking Traces

ARC-AGI Leaderboard

What's new in Claude Opus 5

Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

AI bots ignore evidence. Can we trust them with science?