Does an open-weight decision model beat a hosted one? Jev vs. Laya
6 hours ago
- The author deployed a local model (Laya) for typed decisions in a coding agent, replacing a hosted service (Jev) to keep data local and maintain control.
- Laya provides structured outputs (choice, score, noul) without text generation, using a bidirectional encoder and decision head, avoiding JSON parsing issues.
- The system uses an eligibility gateway to filter models before Laya sees them, and re-validates results, with fallback and abstain options.
- On a held-out replay of 40 routing decisions, Laya achieved 92.5% acceptable decisions vs 82.5% for a deterministic router, but latency varied (p95 from 143 ms to 467 ms on a busy host).
- The author emphasizes local control over managed APIs, with trade-offs in operational responsibility, calibration, and performance.