Turning GLM-5.3-Flash into a Jev-like decision model
11 hours ago
- The technique turns an off-the-shelf LLM (GLM-5.3-Flash) into a Jev-like decision model by obtaining typed decisions with probabilities in a single forward pass, without fine-tuning.
- Key steps: number options, prefill answer with 'choice_index:', then read log probabilities of option tokens using vLLM's logprob_token_ids instead of full JSON generation.
- Evaluation on 29 public datasets shows GLM-5.3-Flash matches Jev in accuracy (median gap 0.7 percentage points, not statistically significant) and outperforms Laya.
- Speed: ~180-300 ms per decision depending on client location; cost: ~EUR 62 per million decisions for GLM-5.3-Flash vs ~EUR 16 for Jev.
- GLM-5.3-Flash supports typed decisions on images (e.g., scanned documents) achieving 70.2% on RVL-CDIP, which Jev and Laya cannot handle.
- Limitations include a maximum 128 logprob tokens per request (requiring two requests for >128 options) and up to 191 single-token option indexes; renaming options affects GLM-5.3-Flash more than others.
- The open-source library and benchmark are available; decisions can be protected by confidential computing via Privatemode.