Replacing Protobuf with Rust to go 5 times faster
5 hours ago
- #Rust
- #PostgreSQL
- #Performance
- PgDog, a PostgreSQL proxy, replaced Protobuf with direct C-to-Rust bindings for better performance.
- The change resulted in a 5x improvement in query parsing and a 10x improvement in deparsing.
- Profiling with samply identified Protobuf as the bottleneck in the pg_query_parse_protobuf function.
- Initial attempts to cache parsed queries worked but had limitations with unique statements and older drivers.
- An LLM (Claude) was used to generate Rust code that maps C types directly to Rust structs, bypassing Protobuf.
- The new implementation uses unsafe Rust for C-to-Rust conversions and recursive algorithms for AST processing.
- Recursive algorithms were chosen for their speed, cache locality, and readability over iterative approaches.
- The change reduced CPU usage and workload, making PgDog faster and cheaper to run.
- The team is looking for a Founding Software Engineer to further develop PostgreSQL horizontal scaling solutions.