Jev Can't Be Calibrated
3 hours ago
- Jev is TypeSafe's System One Model that returns typed decisions with probabilities instead of generating text, useful without training data.
- Jev's calibration claim is questioned because calibration depends on the data distribution, and Jev provides the same probabilities to different datasets.
- Evidence shows Jev can produce miscalibrated probabilities, e.g., predicting 0.92 for a fair coin, and experiments find other models better calibrated.
- To get calibrated probabilities, users should recalibrate Jev on their own data using methods like Platt scaling with a few hundred labeled examples.
- It's recommended to treat Jev's outputs as scores (good for ranking) rather than true probabilities, especially when exact numbers matter.