2 days ago
- Asking an LLM to generate a confidence score for its own response is completely useless and lacks scientific validity.
- Continuous 0-100 confidence scales are problematic; models tend to cluster on round numbers, not use the full range.
- Confidence is ambiguous—it can refer to correctness, coherence, or goal fulfillment—and a single number flattens these distinct concepts.
- LLMs cannot reliably quantify their own confidence due to unreliable internal state understanding and the 'who watches the watchmen' problem.
- Token-level likelihood is not a good proxy for confidence because tokens vary in importance, and collapsing them into one number is not a measurement.
- Research shows self-reported confidence scores are overconfident and require fine-tuning, calibration, or categorical labels to be useful.
- If a model knows it is wrong, it should correct itself rather than output a confidence score; the score serves as a psychological tool, not a reliability measure.
- The real work should focus on improving retrieval quality and search, not on extracting confidence scores from the model.