Fuzzer Blind Spots (Meet Jepsen)
a year ago
- #database
- #fuzz-testing
- #Jepsen
- Systems have components that vary in testability, with deterministic components being easier to test than those with IO and side effects.
- Jepsen testing revealed a correctness bug in TigerBeetle's query engine, a component previously thought to be well-fuzzed.
- The bug involved missing query results when intersecting multiple fields, a scenario not covered by existing fuzzers.
- TigerBeetle's fuzzers missed the bug due to workload constraints that unintentionally avoided probing necessary states.
- The solution involved rewriting the fuzzer to generate random inputs and use a detailed database model for verification.
- The bug was fixed by adjusting the scan logic to handle cases where the target range shrinks during iteration.
- Fuzzers can have blind spots if their workloads unintentionally constrain the state space they explore.
- Complementing targeted fuzzers with broader, simpler approaches can help uncover hidden bugs.