Lessons Learned Building High-Performance Rust Profiler
5 hours ago
- hotpath-rs is an all-in-one Rust profiler/debugging toolkit that combines wall-clock timing, memory allocation, and CPU sampling into unified reports.
- Different performance signals (timing, allocations, CPU) can reveal different bottlenecks, e.g., async sleep dominating wall time but using little CPU.
- The core instrumentation uses RAII guards; performance improved from ~500ns to ~80ns per measurement by batching channel messages to avoid Thread::unpark overhead.
- Memory allocation tracking uses a thread-local stack, but for async functions that may migrate threads, an InstrumentedFuture wrapper is used to correctly attribute allocations per poll call.
- Cache-line contention was reduced by adding #[repr(align(64))] to thread allocation stats, improving multi-threaded performance by up to 86%.
- CPU profiling integrates samply via a separate process; raw CPU traces are resolved by mapping symbol ranges from the binary's debug info.
- Exclusive vs inclusive CPU attribution uses linked-list traversal of stack frames; CPU attribution uses threadCPUDelta rather than sample count to distinguish active CPU time from idle.
- Future work includes DWARF-based symbol resolution, more CPU instrumentation modes, and a hosted version for sharing and comparing profiling reports.