software engineering

Bilingual

LLM's as a Different Kind of Intelligence
3 months ago
- LLMs represent a different kind of intelligence, distinct from human intelligence.
- Human analogies like 'junior engineer' or 'abstraction layer' are flawed because LLMs lack intent and deterministic reasoning.
- LLMs combine vast knowledge with an inability to understand intent, leading to subtle drift from user goals over time.
- Probabilistic behavior from LLMs challenges the deterministic foundations of software engineering.
- Effective use requires learning LLMs' unique failure modes rather than forcing familiar comparisons.
Model Training as Code
3 months ago
- Model training complexity requires specialized teams and manual coordination is inefficient.
- Aleph Alpha's Savanna implements Model Training as Code (MTaC) for hermetic, one-click training runs.
- Manual training leads to human errors, lost learnings, and fragmented team ownership.
- MTaC offers composability, consensus via version control, and provenance through code history.
- Savanna uses trunk-based development, CI for validation, and immutable artefact versioning.
- It enables hyperparameter sweeps with caching to avoid redundant computations.
- The system runs on Flyte in Kubernetes, with artefact tracking and real-time monitoring.
- MTaC improves iteration speed, facilitates team restructuring, and supports auto-research.
Most rewrites serve the engineer, not the business
3 months ago
- Rewrites often prioritize the engineer's preferences over business value, as shown by an example of porting a CakePHP application to Laravel without tangible benefits.
- Existing code holds historical knowledge and bug fixes; rewriting it risks reintroducing past issues and losing valuable 'scar tissue' documented by Joel Spolsky.
- Unfamiliarity with a codebase or framework can lead to misjudgments; assuming previous developers made reasoned choices is crucial before deeming code 'broken'.
- Legitimate reasons for rewrites include end-of-life runtimes, loss of key personnel, high feature development costs, or business needs that outgrow the current design.
- AI can accelerate code generation but may omit context for past fixes, leading to rediscovery of old bugs and perpetuating a cycle of rewrites without addressing root causes.
- To break the rewrite cycle, document the rationale behind code decisions clearly, ensuring future engineers or AI models understand the 'why' and not just the 'what'.
Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
3 months ago
- Senior SWE-Bench treats agents like senior engineers by using realistic, natural language instructions instead of over-specified requirements.
- It introduces a validation agent that writes behavioral tests to evaluate tasks, adapting to submitted solutions.
- Bug tasks are based on tricky user reports requiring runtime investigation, such as debugging logs and reproduction steps.
- Scoring combines runtime correctness tests with quality metrics to assess tasteful code solutions.
- Tasks are sourced from PRs in diverse repositories and involve multi-phase, multi-stack features or bugs with significant runtime investigation.
- Instructions are naturally under-specified, with a median length 31% that of SWE-Bench Pro.
- Feature tasks can span multiple services, averaging 11 files touched per task, and are long-horizon, requiring hundreds of steps.
- A leaderboard shows top-performing models like Claude Opus 4.8 achieving a 24.0% solve rate, with frontier models failing over 75% of the time.

software engineering

LLM's as a Different Kind of Intelligence

Model Training as Code

Most rewrites serve the engineer, not the business

Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers