OpenJev
6 hours ago
- Direct readout computes probabilities from choice logits, normalizing only over supplied options without decoding.
- The tool offers a live local experiment where a model can either read probabilities or generate them token-by-token as JSON.
- Choose a model size (e.g., MiniCPM5 2B or Qwen3 0.6B for low-end devices) and run both methods on your own hardware to compare.
- Included model accuracy table shows Qwen3 0.6B, MiniCPM5 2B, and Qwen3.5 4B with stats on authored (balanced accuracy), perturbed, and TypeSafe (equal-case agreement) metrics.
- Weights load from Hugging Face and stay in browser cache; inputs never leave the page, but first load may take minutes depending on model, network, and GPU.
- The decision process uses the same input for both paths: one reads probabilities directly, the other asks for JSON text generation.
- 02A section explains direct readout normalizes over only displayed options and is not calibrated confidence.
- 02B section details generation path asking the model to output its estimated distribution as JSON.
- Methods run sequentially on the same model to avoid GPU contention; direct runs first, then generation.
- Timing for setup, warmup, prompt preparation, direct execution, and generation is measured with performance.now() – no canned results.
- The demo uses quantized GGUF weights via wllama, which can affect quality and speed; phone model trades accuracy for size, and the 4B option requires more memory.