- Non-deterministic bugs are harder to debug because they require repeated execution and often involve data corruption that occurs before the crash.
- The bug was discovered in the redis-client library after a Redis gem upgrade caused flaky tests in Buildkite's CI system.
- The team used Test Engine to correlate test reliability, core dumps to inspect crash states, and ASAN to identify a heap use-after-free in hiredis C extension.
- The fix involved disabling the C extension for Redis connections and reporting the issue upstream.
- The debugging process combined skill, teamwork, and luck, with tools enabling detection of memory corruption that would otherwise take months to fix.