AI Has Passed Every Exam. It Has Never Had an Idea

5 hours ago

AI excels at passing exams and finding answers, achieving superhuman results on tests like Humanity's Last Exam and the International Math Olympiad.
AI contributes to scientific discoveries (e.g., AlphaFold, GNoME) but only when humans decide which questions to ask; it has not independently chosen important questions.
The contradiction between AI's superb answer-finding and lack of question-posing is resolved by recognizing that all benchmarks measure only answer-finding, not question-choosing.
Historical breakthroughs like Einstein's theory of relativity came from questioning existing assumptions, not just better answer-finding.
Tests like the proposed 'Einstein test' still measure answer-finding because they pre-select the question and data, missing the essence of genius.
Our inability to measure curiosity or question-choosing stems from a long-standing focus on grading answers, which penalizes or ignores question-posing.
The machine's intelligence is limited to the articulable, measureable part of human intelligence, while the unmeasured part—the choice of questions—remains silent.
Without instruments to detect question-asking, we mistake our blind spot for a limit of machines.

AI chatbots like ChatGPT, Gemini, and Grok failed to update predictions based on experimental evidence in a pen demonstration, sticking to incorrect assumptions.
A study showed AI agents ignored evidence in 68% of scientific reasoning tasks, made unsupported claims in 53%, and used contradictory evidence to change output only 26% of the time.
AI systems lack an iterative reasoning process similar to human scientists, often refusing to revise hypotheses despite clear evidence, limiting their reliability in science and medicine.

The most consequential AI question is how we will know what superintelligent machines are thinking, not when they will arrive.
Analogies from 'Arrival' and 'Project Hail Mary' illustrate the challenge of understanding cognitively alien intelligences.
Current AI systems are fluent but structurally alien; we lack calibrated instruments for evaluation.

AI possesses infinite objective intelligence (e.g., facts, calculations) but lacks subjective intelligence (wisdom, taste, genuine understanding).
Objective intelligence cannot be traded for subjective intelligence; they are distinct and non-fungible, unlike resources in Settlers of Catan.
AI writing is technically perfect but emotionally hollow, missing the 'light' that makes prose alive, as consensus among writers shows.

Kepler discovered planetary motion laws through a blend of creativity and data analysis, not solely through genius.
AI reduces the cost of generating scientific ideas, shifting the bottleneck to verification and validation.
The verification loop for correct scientific theories can span decades, often with initially worse predictions.

AI predictions from 2023 were largely accurate but underestimated the ensuing hype magnitude.
LLMs are not truly intelligent but perform 'absolute miracles' in tasks like grammar correction and code debugging.
The Chinese Room Argument remains relevant for debating AI intelligence vs. simulation.

AI chatbots like ChatGPT, Gemini, and Grok failed to update predictions based on experimental evidence in a pen demonstration, sticking to incorrect assumptions.
A study showed AI agents ignored evidence in 68% of scientific reasoning tasks, made unsupported claims in 53%, and used contradictory evidence to change output only 26% of the time.
AI systems lack an iterative reasoning process similar to human scientists, often refusing to revise hypotheses despite clear evidence, limiting their reliability in science and medicine.

The most consequential AI question is how we will know what superintelligent machines are thinking, not when they will arrive.
Analogies from 'Arrival' and 'Project Hail Mary' illustrate the challenge of understanding cognitively alien intelligences.
Current AI systems are fluent but structurally alien; we lack calibrated instruments for evaluation.

AI possesses infinite objective intelligence (e.g., facts, calculations) but lacks subjective intelligence (wisdom, taste, genuine understanding).
Objective intelligence cannot be traded for subjective intelligence; they are distinct and non-fungible, unlike resources in Settlers of Catan.
AI writing is technically perfect but emotionally hollow, missing the 'light' that makes prose alive, as consensus among writers shows.

Kepler discovered planetary motion laws through a blend of creativity and data analysis, not solely through genius.
AI reduces the cost of generating scientific ideas, shifting the bottleneck to verification and validation.
The verification loop for correct scientific theories can span decades, often with initially worse predictions.

AI predictions from 2023 were largely accurate but underestimated the ensuing hype magnitude.
LLMs are not truly intelligent but perform 'absolute miracles' in tasks like grammar correction and code debugging.
The Chinese Room Argument remains relevant for debating AI intelligence vs. simulation.

Hasty Briefsbeta