What's the largest software project AI can complete on its own?

4 hours ago

MirrorCode is a benchmark co-developed with METR for long-horizon AI coding tasks, requiring complete end-to-end reimplementation of programs without access to the original source code.
The benchmark features 25 target programs spanning Unix utilities, bioinformatics, cryptography, interpreters, compression, and other domains.
It provides scale-aware evaluations with realistic inference budgets (up to $2,600 and 19 days per run), and is sandboxed and cheat-resistant with held-out tests.
Frontier models can solve complex tasks; for example, Claude Opus 4.7 reimplemented the gotree bioinformatics toolkit in 14 hours at a cost of $251, a task estimated to take a human engineer 2–17 weeks.
Data contamination is a caveat since the programs are open source, but performance on memorization screens suggests memorization is not the dominant factor.
The leaderboard includes 30 tasks from the Medium and Large difficulty buckets, run with a budget of 10 billion tokens per attempt, and the benchmark is not fully solved.
The scaffold and 22 out of 25 MirrorCode target programs are released as open source.

Inkling is a multimodal autoregressive transformer model with 975B total parameters (41B active) released by Thinking Machines Lab on July 15, 2026, under Apache 2.0 license.
It accepts text, image (40px-4096px), and audio (16kHz WAV up to 20 mins) inputs, generating text outputs in English and multiple languages, with a 1M token context window.
Intended for developers building AI applications like chatbots, coding assistants, and agentic systems, it's available via API (Tinker/third-party) and open weights on Hugging Face.

Senior SWE-Bench treats agents like senior engineers by using realistic, natural language instructions instead of over-specified requirements.
It introduces a validation agent that writes behavioral tests to evaluate tasks, adapting to submitted solutions.
Bug tasks are based on tricky user reports requiring runtime investigation, such as debugging logs and reproduction steps.

BTL-3 is a 27B open-weight agent model designed for agentic coding, structural tool use, repository work, failure recovery, and long multi-turn execution.
It achieves high scores on benchmarks: 95.1% HumanEval pass@1, 88.5% BFCL v4 AST, and 88.1% LiveCodeBench v6.
A Compact edition (8.39 GB) packages the full model into a single file with 92.2% behavior retention, enabling native local inference.

Motif-3-Beta is a preview checkpoint of the upcoming Motif-3 Mixture-of-Experts (MoE) language model, not the final release.
It has approximately 314 billion total parameters with only about 13 billion active per token, using sparse MoE with 384 experts (8 activated per token plus 1 shared expert).
Supports a native context length of 256K tokens (262,144) and is built for multilingual, general-purpose tasks.

Laguna S 2.1 is a 118B-parameter Mixture-of-Experts model with 8B active parameters per token and a 1M-token context window, developed in under nine weeks.
It achieves competitive scores against much larger models on long-horizon coding benchmarks, scoring 70.2% on Terminal-Bench 2.1 and 40.4% on DeepSWE v1.1.
All evaluation trajectories are publicly released at trajectories.poolside.ai for transparency and reproducibility.

Inkling is a multimodal autoregressive transformer model with 975B total parameters (41B active) released by Thinking Machines Lab on July 15, 2026, under Apache 2.0 license.
It accepts text, image (40px-4096px), and audio (16kHz WAV up to 20 mins) inputs, generating text outputs in English and multiple languages, with a 1M token context window.
Intended for developers building AI applications like chatbots, coding assistants, and agentic systems, it's available via API (Tinker/third-party) and open weights on Hugging Face.

Senior SWE-Bench treats agents like senior engineers by using realistic, natural language instructions instead of over-specified requirements.
It introduces a validation agent that writes behavioral tests to evaluate tasks, adapting to submitted solutions.
Bug tasks are based on tricky user reports requiring runtime investigation, such as debugging logs and reproduction steps.

BTL-3 is a 27B open-weight agent model designed for agentic coding, structural tool use, repository work, failure recovery, and long multi-turn execution.
It achieves high scores on benchmarks: 95.1% HumanEval pass@1, 88.5% BFCL v4 AST, and 88.1% LiveCodeBench v6.
A Compact edition (8.39 GB) packages the full model into a single file with 92.2% behavior retention, enabling native local inference.

Motif-3-Beta is a preview checkpoint of the upcoming Motif-3 Mixture-of-Experts (MoE) language model, not the final release.
It has approximately 314 billion total parameters with only about 13 billion active per token, using sparse MoE with 384 experts (8 activated per token plus 1 shared expert).
Supports a native context length of 256K tokens (262,144) and is built for multilingual, general-purpose tasks.

Laguna S 2.1 is a 118B-parameter Mixture-of-Experts model with 8B active parameters per token and a 1M-token context window, developed in under nine weeks.
It achieves competitive scores against much larger models on long-horizon coding benchmarks, scoring 70.2% on Terminal-Bench 2.1 and 40.4% on DeepSWE v1.1.
All evaluation trajectories are publicly released at trajectories.poolside.ai for transparency and reproducibility.

Hasty Briefsbeta