Bilingual

Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
4 days ago
- Senior SWE-Bench treats agents like senior engineers by using realistic, natural language instructions instead of over-specified requirements.
- It introduces a validation agent that writes behavioral tests to evaluate tasks, adapting to submitted solutions.
- Bug tasks are based on tricky user reports requiring runtime investigation, such as debugging logs and reproduction steps.
- Scoring combines runtime correctness tests with quality metrics to assess tasteful code solutions.
- Tasks are sourced from PRs in diverse repositories and involve multi-phase, multi-stack features or bugs with significant runtime investigation.
- Instructions are naturally under-specified, with a median length 31% that of SWE-Bench Pro.
- Feature tasks can span multiple services, averaging 11 files touched per task, and are long-horizon, requiring hundreds of steps.
- A leaderboard shows top-performing models like Claude Opus 4.8 achieving a 24.0% solve rate, with frontier models failing over 75% of the time.
Most rewrites serve the engineer, not the business
4 days ago
- Rewrites often prioritize the engineer's preferences over business value, as shown by an example of porting a CakePHP application to Laravel without tangible benefits.
- Existing code holds historical knowledge and bug fixes; rewriting it risks reintroducing past issues and losing valuable 'scar tissue' documented by Joel Spolsky.
- Unfamiliarity with a codebase or framework can lead to misjudgments; assuming previous developers made reasoned choices is crucial before deeming code 'broken'.
- Legitimate reasons for rewrites include end-of-life runtimes, loss of key personnel, high feature development costs, or business needs that outgrow the current design.
- AI can accelerate code generation but may omit context for past fixes, leading to rediscovery of old bugs and perpetuating a cycle of rewrites without addressing root causes.
- To break the rewrite cycle, document the rationale behind code decisions clearly, ensuring future engineers or AI models understand the 'why' and not just the 'what'.
Model Training as Code
10 days ago
- Model training complexity requires specialized teams and manual coordination is inefficient.
- Aleph Alpha's Savanna implements Model Training as Code (MTaC) for hermetic, one-click training runs.
- Manual training leads to human errors, lost learnings, and fragmented team ownership.
- MTaC offers composability, consensus via version control, and provenance through code history.
- Savanna uses trunk-based development, CI for validation, and immutable artefact versioning.
- It enables hyperparameter sweeps with caching to avoid redundant computations.
- The system runs on Flyte in Kubernetes, with artefact tracking and real-time monitoring.
- MTaC improves iteration speed, facilitates team restructuring, and supports auto-research.
LLM's as a Different Kind of Intelligence
16 hours ago
- LLMs represent a different kind of intelligence, distinct from human intelligence.
- Human analogies like 'junior engineer' or 'abstraction layer' are flawed because LLMs lack intent and deterministic reasoning.
- LLMs combine vast knowledge with an inability to understand intent, leading to subtle drift from user goals over time.
- Probabilistic behavior from LLMs challenges the deterministic foundations of software engineering.
- Effective use requires learning LLMs' unique failure modes rather than forcing familiar comparisons.
Zero-defects code: the prescient Microsoft memo from 1989
5 hours ago
- Microsoft's 1989 Zero Defects memo advocated for principles like continuous integration and test-driven development long before they became mainstream.
- The memo highlighted problems in Microsoft's culture, such as rewarding developers for checking in incomplete code and treating schedules as more important than quality.
- Chris Mason argued that writing perfect, zero-defects code was achievable and emphasized the need for a mindset shift to prioritize quality over speed.
- Key solutions proposed included having a working, shippable product daily, fixing bugs immediately, and shifting testing to quality assurance rather than bug-finding.
- The memo suggested practical techniques like code reviews, writing tests before debugging, and using asserts to verify assumptions, aiming to reduce bugs and improve predictability.
- Leadership committed to supporting realistic schedules and pushing back against pressures to cut corners, emphasizing that taking minor heat early was better than major issues later.

Hasty Briefsbeta

#software engineering

Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

Most rewrites serve the engineer, not the business

Model Training as Code

LLM's as a Different Kind of Intelligence

Zero-defects code: the prescient Microsoft memo from 1989