9 hours ago
- A long-running process crash is problematic, especially when it holds significant memory state; web programming frameworks that create new interpreters per page view offer a more reliable paradigm.
- Redis experienced a crash in its radix tree implementation, with error addresses zero-padded at the end of the heap, indicating a read overflow beyond memory bounds.
- Extensive fuzz testing failed to reproduce the crash because the bug was already fixed in a newer version of the code, but not backported to Redis 4.0 due to being bundled in a Streams commit.
- The crash analysis involved examining CPU registers and disassembly, revealing that the compiler-generated code did not match the C source due to version mismatch.
- Key lessons include the value of detailed crash reports for debugging system software, the necessity of understanding AMD64 assembler, the power of fuzz testing with reference comparisons, and careful management of bug fixes across branches.
- Seeking help from knowledgeable peers (e.g., Fedor Indutny) was crucial in resolving the mystery.