The dead weight loss of strictly isotonic regression
3 days ago
- #model evaluation
- #calibration
- #isotonic regression
- Calibration aligns model scores with event frequencies, estimating the calibration function on a holdout set.
- Isotonic regression merges neighboring regions to enforce nondecreasing calibration, often creating ties and removing local distinctions.
- Strict isotonic regression can flatten distinctions due to sampling variability, reducing resolution and granularity.
- Noise-based flattening is desirable when adjacent scores have similar risks, while limited-data flattening is undesirable.
- Diagnostics include examining stability, evaluating conditional AUC among tied pairs, and tracking unique calibrated values with sample size.
- Calibre provides monotone calibrators that avoid unnecessary ties, using methods like nearly-isotonic regression and relaxed PAVA.
- Evaluation focuses on achieving strong calibration while preserving resolution, avoiding flattening due to limited sample size.