Test-case reducers are underappreciated debugging tools
11 hours ago
- #debugging
- #test-automation
- #software-tools
- Test-case reducers are tools that automatically reduce the size of a failing input while preserving the error through an 'interestingness test'.
- They are not magic; a simple reducer can be written in Python to iteratively remove parts of the input and check if the error persists with a shell script.
- Reducers like Shrink Ray are more advanced, using clever techniques (e.g., removing comments, reducing integers) and parallelism for efficient reduction.
- Interestingness tests must be precise to avoid over-reduction, fast to speed up the process, and handle issues like nondeterminism and timeouts.
- Reducers can be abused to reduce more than just input length, such as by using a global counter to minimize trace size or error frequency.
- For nondeterministic bugs, test-case reducers can increase error frequency or determinism by adjusting interestingness tests to run inputs multiple times.
- Test-case reducers are valuable for debugging beyond compilers, making bugs easier to understand by producing minimal, focused inputs.