Detect memory leaks of C extensions with psutil and psleak
4 months ago
- #Python
- #Cextensions
- #MemoryLeaks
- Memory leaks in Python C extensions are hard to diagnose with traditional metrics like RSS and VMS.
- psutil 7.2.0 introduces new APIs (heap_info() and heap_trim()) for C heap introspection to detect native leaks.
- These APIs bypass Python's memory management, providing direct insight into the platform's native allocator.
- Many Python projects rely on C extensions (NumPy, pandas, PyTorch, etc.), making native leak detection crucial.
- heap_info() provides metrics like heap_used and mmap_used to track C-level memory allocations.
- heap_trim() helps reduce allocator noise for cleaner leak detection.
- A new tool, psleak, automates leak detection by running functions repeatedly and tracking memory growth.
- psleak is integrated into psutil's test suite to ensure no memory leaks in its C code.
- Monitoring the C heap is essential for detecting leaks that don't affect Python's reference counts or RSS.
- psutil 7.2.0 enhances observability, turning it into a debugging tool for C extension leaks.