a day ago
- Binary IQ refers to an LLM's ability to correctly answer simple yes/no questions when all relevant information is in the context window, and the author hypothesizes this is the main bottleneck for reliable agents.
- Best current LLMs (e.g., GPT-4o, Claude 3.5) show high in-distribution IQ (~125) but much lower out-of-distribution IQ (~75), making agents unreliable despite high average success rates.
- Collective IQ suggests that groups of average-IQ individuals or models cannot exceed the decision-making capability of the smartest member, analogous to LLM agents being unable to surpass the intrinsic binary IQ of the underlying network.
- Techniques like chain-of-thought or recursive checking help but do not fundamentally raise binary IQ, as every step still requires a final binary decision.
- The author proposes evaluating binary IQ using re-skinned boolean algebra expressions of increasing complexity to measure out-of-distribution reasoning.
- LLMs are not yet deeply smart enough for robust agent tasks, though future model generations might improve this.