Making PyPI's test suite 81% faster
a year ago
- #Python
- #Performance
- #Testing
- Trail of Bits collaborated with PyPI to improve test suite performance, reducing execution time from 163 seconds to 30 seconds.
- Key optimizations included parallelizing test execution with pytest-xdist (67% reduction), using Python 3.12’s sys.monitoring for coverage (53% reduction), optimizing test discovery, and removing unnecessary imports.
- The test suite grew from 3,900 to 4,700 tests while achieving an 81% performance improvement.
- Parallelization required solutions for database fixtures, coverage reporting, and test output readability.
- Optimizing coverage with sys.monitoring in Python 3.12 further reduced test execution time by 53%.
- Strategic use of testpaths configuration reduced test discovery time by 66%.
- Removing unnecessary imports like ddtrace provided a modest 3.4% improvement.
- A proof-of-concept for database migration squashing showed potential but was not merged due to complexity.
- Faster tests improve security by encouraging frequent and thorough testing.
- Quick wins for other projects include parallelizing tests, optimizing coverage, speeding up test discovery, and eliminating unnecessary imports.