A 40ms Go GC pause caused by swap
2 days ago
- A Go GC stop-the-world pause increased from a median of 51 microseconds to 40 milliseconds due to metadata pages in swap, causing a significant performance issue.
- The root cause is that the kernel evicts rarely accessed pages to swap, including the GC's metadata, which is read during the pause, leading to major page faults.
- During the worst pause, 228 page faults accounted for 39 of the 40 milliseconds, with the GC's bookkeeping operations like spanSet.reset and finishsweep_m being the culprits.
- The problem occurs during both sweep and mark terminations, with 312 such pauses in 30 minutes during the test, highlighting the frequency of the impact.
- The pause affects all goroutines and stops I/O handling, making 40ms a critical latency issue, especially for production services.
- Additionally, building a 511 KiB message took 105ms on NVMe and 903ms on network storage, a secondary cost impacting the allocating goroutine only.
- The author agrees with Chris Down that swap isn't evil but warns that it interacts poorly with garbage collection, and the impact persists even with Go 1.26's Green Tea GC.
- Detailed experiments, including BPF scripts and plots, are publicly available on GitHub for further analysis.