The Null Is Always False (Except When It Is True) (2014)
3 days ago
- #Research Methods
- #Frequentist Statistics
- #Null Hypothesis Significance Testing
- The null hypothesis assumes zero difference between population means, but sample means vary, and differences approach zero with infinite samples, a core Frequentist assumption.
- Cohen (1990) argues the null hypothesis is always false in the real world, rejecting it with large samples is trivial, but requires immense sample sizes for tiny effects (e.g., d=0.001 needs ~31 million participants).
- Rejecting the null doesn't confirm if an effect is stable or diminishes with more data; without significance, it's unclear if it's a Type 2 error or a true null.
- The population is dynamic (e.g., births/deaths), so exact null truth at any moment isn't critical for generalizable statements in psychology.
- Under true null with randomization, p-values are uniformly distributed, but in correlational research without random assignment, the null is rarely true due to universal correlations (Meehl's crud factor).
- Analysis of Many Labs data (n=6344) shows random assignment yields non-significant results (null works), while gender (a non-manipulated variable) shows multiple small but significant effects, highlighting NHST's limitations in explaining causes.
- NHST can reject the null but is inconclusive for non-significant results; it's useful in experiments with random assignment but requires better models for deeper understanding.
- Debate includes whether to assume a null (even if approximating zero) or estimate effects directly, with Bayes factors comparing hypotheses' predictive success rather than absolute truth.