Ten ways a check passes while the thing it checks is broken
3 hours ago
- A check that has never failed is unproven—break the thing deliberately to confirm it can fail.
- Check at the layer where the consequence happens (e.g., rendered page, not just source) because easy checks miss failures.
- Reliability is not validity: a measure can be consistent but measure the wrong thing; test invariance under irrelevant transformations.
- Test data that knows the hypothesis is weak; use material that predates the question to avoid confirmation bias.
- Design experiments to distinguish the effect from its mimic, not just to detect an effect.
- Silence is ambiguous: absence of an error is not presence of success; require a positive signal (e.g., 201, stored record).
- Assert from the claim (e.g., sentence in docs) not from the behavior, because behavior can be fine while the claim drifts.
- Verify the immutable artifact (deployment URL, commit) rather than a friendly alias that may serve a cached or old copy.
- Let tools refuse to silently succeed: prefer editors that error on unmatched patterns, and assert match counts for bulk changes.
- Keep a human in the loop—the only instrument with no fixed dialect can spot what automated checks miss.