Reinforcement Learning with Metacognitive Feedback Elicits Uncertainty in LLMs
7 hours ago
- #Reinforcement Learning
- #Uncertainty Calibration
- #LLM Metacognition
- Large Language Models (LLMs) often show poor metacognition, leading to overconfidence, hallucinations, and failure to recognize their own knowledge limits.
- The paper introduces Reinforcement Learning with Metacognitive Feedback (RLMF), a method that uses a model's self-judgments of performance to improve completion rankings during preference optimization and for selecting high-value training data.
- A second method, metacognitive data selection, uses similar self-judgments to outperform standard active learning by identifying more useful training examples.
- These innovations are applied to the task of faithful calibration (FC), aiming to align an LLM's expressed uncertainty with its actual intrinsic uncertainty.
- A two-stage approach is used: first calibrating self-reported confidence scores for faithfulness, then mapping these to natural, context-adaptable language through output editing.
- Extensive experiments demonstrate that RLMF achieves state-of-the-art performance in faithful calibration across diverse tasks without sacrificing accuracy, and surpasses standard reinforcement learning by up to 63%.
- RLMF enhances LLMs' ability to assess and express their own capability limits, suggesting it as a promising paradigm to improve LLM metacognition, leading to better capabilities and alignment.