How AST-grep Rewrote Tree-sitter in Rust and Made It 30% Faster
5 hours ago
- ast-grep rewrote Tree-sitter's C core in Rust using AI, resulting in a 30% faster parser and 22% faster end-to-end performance in ast-grep.
- The Rust rewrite preserved existing grammar compatibility and binary interfaces while removing incremental parsing and native WebAssembly language loading.
- Performance gains came from optimizing GLR parsing (using a linear stack for common cases), memory allocation (arena-based), and tree traversal (caching child lookups).
- Memory usage increased as a trade-off, peaking at 91.2 MiB on a large corpus, down from over 1 GiB earlier in the project.
- End-to-end application testing revealed that parser-only benchmarks missed overheads from tree traversal and memory allocation, which were later resolved.
- The project involved iterative AI collaboration, with the author directing ChatGPT to profile, refactor, and optimize while validating through tests and real workloads.