Just make it scale: An Aurora DSQL story
a year ago
- #Rust
- #Database Engineering
- #Aurora DSQL
- Aurora DSQL was announced at re:Invent, focusing on scaling database engineering with a transition from JVM to Rust.
- The development journey of DSQL involved deep technical insights into Rust, engineering efficiency, and questioning past decisions.
- AWS's database portfolio evolved from Amazon RDS to purpose-built options like DynamoDB, Redshift, and Aurora, addressing varied customer needs.
- Aurora DSQL's architecture breaks the database into modular components, each handling specific tasks efficiently while working together.
- Scaling writes horizontally was a challenge; DSQL adopted a single journal approach for commits, simplifying writes but complicating reads.
- The Crossbar component was introduced to manage read and write paths separately, addressing network bandwidth limitations.
- Initial simulations revealed performance issues with 40 hosts, leading to a decision to switch from JVM to Rust for better performance and memory safety.
- Rust's predictable performance, memory safety, and zero-cost abstractions made it ideal for DSQL's data plane.
- The team started with rewriting the Adjudicator in Rust, achieving a 10x performance improvement over the Kotlin version.
- PostgreSQL was chosen for query processing, with extensions written in Rust to ensure memory safety and leverage existing libraries.
- The control plane, initially written in Kotlin, was later rewritten in Rust to unify logic and improve integration with the data plane.
- Rust's adoption led to consistent tail latencies and predictable performance, making it a strong fit for DSQL's requirements.
- The team emphasized learning and development, with internal guides and sessions to build expertise in Rust and distributed systems.
- Rust's productivity matched Java once the team was ramped up, though it may not be suitable for every project.
- The decision to use Rust was based on specific requirements, team capabilities, and operational environment, highlighting the importance of thoughtful architectural choices.