When LLM judges agree, should we believe them?
a day ago
- Evaluating a retrieval-augmented-generation system with multiple LLM judges: 8 say relevant, 2 say not; majority vote seems convincing, but independence of judges is critical.
- If agreeing judges share prompts, training, model family, or blind spots, their agreement may be misleading; vote count can overstate evidence.
- The paper 'Dependence-aware label aggregation for LLM-as-a-judge via Ising models' addresses this by modeling judge correlations to adjust aggregate scores for diversity.
- The method, tested on three tasks, outperforms weighted majority vote by 9% to 14% on standard metrics.
- Majority vote assumes independent errors, which is often unrealistic for LLM judges; shared biases can make majorities less informative.
- A judge panel is modeled as a network using an Ising model, learning both judge reliability and pairwise dependence, without needing human labels.
- Two variants: a simpler model with uniform dependence across labels, and a class-dependent model that captures different agreement patterns for positive vs. negative labels.
- The algorithm iteratively estimates true labels and judge parameters from vote patterns, useful for teams with existing LLM-as-a-judge logs.
- Best practices include evaluating the whole panel for redundancy, treating model diversity as statistical diversity, inspecting agreement structure, and reporting dependence-aware uncertainty.
- Agreement among LLM judges requires scrutiny: it could be independent evidence or a shared blind spot; good aggregation distinguishes them.