A single function Jev-like wrapper for LLMs, including vision models
3 hours ago
- The author discovers the trick of reading an LLM's token probabilities by forcing a one-token output and using the logprobs parameter.
- This technique enables fast classification by posing multiple-choice questions and interpreting the model's confidence via log probabilities.
- It works with vision models by attaching images; example: analyzing webcam frames for person presence, indoor/outdoor status, and brightness.
- Performance: ~1 frame per second with Gemma 4 12B on an RTX 3090, ~0.2 FPS with OpenAI's API, using three questions per frame.
- A Python script uses OpenCV for webcam capture and llama.cpp or OpenAI API to send requests and decode logprobs.
- Setup instructions include downloading the Gemma 4 model and llama.cpp binary, then running the script locally or with OpenAI.
- The approach offers flexibility: conditions can be changed by editing plain-text questions, unlike specialized computer vision models.