Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
3 days ago
- Real-SWE is a benchmark for evaluating AI models on private, real-world enterprise codebases, with tasks derived from actual production problems.
- Tasks involve proprietary systems not available publicly, requiring understanding of business logic, company-specific conventions, and cross-service changes.
- Top models achieve resolution rates between 16-38%, with Fable 5.1 leading at 38.8% and GPT-5.6 Sol lowest at 16.2%.
- Codebases are selected from real companies with significant usage, including a consumer fintech platform and an enterprise AI sales platform.
- Prompts are slightly underspecified, relying on agents to discover implementation details in the codebase and tools like AWS, Kubernetes, and various databases.
- Common failure modes include missed requirements, with 71.4% of short rollouts failing; costs range from $2.50 to $6.96 per rollout.
- Evaluation uses native harnesses and verifiers, with tasks spanning areas like billing, tax jurisdiction, and data migrations.
- The benchmark highlights gaps in model ability to handle company-specific engineering patterns and verify assumptions, as seen in low pass rates on complex tasks.