Making ast.walk 220x Faster
3 hours ago
- #Performance tuning
- #AST traversal
- #Python optimization
- ast.walk is initially slow, especially in linting massive generated Python code.
- Generators (yield) in ast.walk cause performance issues due to execution suspension in hot paths.
- Inlining iter_child_nodes and iter_fields improves performance by removing generator overhead.
- Using getattr(node, field, None) instead of yielding tuples further speeds up the process.
- Combining field reading and subclass checking in one call adds incremental improvements.
- Rewriting the logic in Rust allows significant performance gains via native machine code.
- Iterating over __dict__ directly and caching subclass information reduces dictionary overhead.
- Precomputing AST subclass data into a cache-friendly table improves speed via L1 cache usage.
- Focusing only on _fields entries avoids checking unnecessary attributes like lineno/col_offset.
- Final optimizations achieve ~220x faster performance compared to original ast.walk.