Leak and Seek a Go Runtime Mystery
a year ago
- #Memory Leak
- #Debugging
- #Go
- A memory leak issue was reported by the customer success team, leading to significant performance problems for large customers.
- Initial investigation ruled out goroutine leaks but pointed to issues within the SQLite3 driver based on heap profiles.
- The team suspected finalizers in Go's runtime might be causing circular references, but detailed examination found no such dependency.
- Further debugging suggested a potential Go runtime bug, particularly with the finalizer goroutine being blocked.
- Using goref, the team confirmed that no live objects referenced the leaked data, pointing again to a runtime issue.
- The root cause was identified as a blocking bug in the go-smb2 package, which blocked the finalizer goroutine.
- A debugging anomaly was discovered where mfinal.go traces were missing in certain debug profiles, later reported to the Go team.
- The team implemented a monitoring solution to detect blocked finalizers in real-time, visualized via Grafana.
- The investigation highlighted the importance of persistence, teamwork, and collaboration in solving complex technical issues.
- The findings contributed to the Go community, including discussions on runtime improvements and debugging fixes.