Learnings from 100K lines of Rust with AI (2025)
11 hours ago
- #AI-Assisted Development
- #Rust
- #Distributed Systems
- The author stress-tested AI coding agents by building a modern, production-grade Rust-based multi-Paxos consensus engine, achieving 100K lines of Rust in ~4 weeks and improving performance from 23K to 300K ops/sec in ~3 weeks.
- Modernizing Azure's Replicated State Library (RSL) was driven by gaps like lack of pipelining, no NVM support, and limited hardware awareness (e.g., RDMA), aiming for lower latency and higher throughput for modern cloud and AI workloads.
- AI coding agents, especially Claude Code and Codex CLI, boosted productivity, enabling implementation of over 130K lines of Rust covering multi-Paxos, leader election, log replication, snapshotting, and configuration changes in ~6 weeks.
- Code contracts, generated and refined by AI, ensured correctness by specifying preconditions, postconditions, and invariants, with AI also generating tests and property-based tests to catch bugs, like a subtle Paxos safety violation.
- A lightweight spec-driven development approach replaced rigid SDD, using AI to generate specs, critique user stories, and plan features, with a focus on single user stories as manageable units for AI agents.
- Aggressive performance optimization involved AI in instrumenting metrics, analyzing bottlenecks, and proposing optimizations (e.g., reducing lock contention and allocations), leveraging Rust's safety for confidence in changes.
- A wish list for AI-assisted coding includes end-to-end user story execution with more AI autonomy, automated contract workflows for test generation and debugging, and autonomous performance optimization for larger codebases.
- The project, based on a design by Jay Lorch, has addressed pipelining and NVM support (using a verified persistence log), with RDMA support pending, and includes over 1,300 tests accounting for more than 65% of the codebase.