Show HN: Jevstiller – Distill Jev into a local model, with a disagreement bound
6 hours ago
- Jevstiller trains a lightweight local model (linear layer over frozen sentence encoder) to mimic a larger model Jev, with a contractual guarantee that it agrees with Jev on at least a specified percentage of requests (e.g., 98%).
- The router decides whether to answer locally (≈15 ms on CPU) or fall back to Jev (≈300 ms) using a confidence threshold and OOD detection, calibrated with a statistically rigorous bound (Clopper-Pearson and fixed-sequence testing) that holds with 95% probability.
- A simple point-estimate threshold often exceeds the 2% disagreement budget due to noise in calibration data; Jevstiller's bound method trades 4–8% coverage for a reliable guarantee.
- The system also respects Jev's uncertainty: if Jev would be unsure (below a confidence floor), a local answer counts as disagreement, ensuring the same budget covers both label mismatches and uncertainty flags.
- A permanent audit stream (2% of traffic) monitors live agreement, detects drift, and triggers automatic retraining—e.g., recovering from a teacher change in under an hour.