Epicycles All the Way Down
4 days ago
- #understanding vs compression
- #pattern recognition
- #LLM limitations
- The author recounts personal anecdotes about relying on re-derivation over memory in exams and attempting to learn poker through outcomes rather than calculation, illustrating a flawed approach to knowledge acquisition.
- LLMs are described as pattern-fitters that often overfit data, learning surface patterns rather than underlying generative principles, leading to failures that resemble market crashes rather than systematic errors.
- Mathematically, there are more ways to generate patterns than patterns themselves, and LLMs struggle to identify the simplest true generator due to inductive biases and constraints like Gold's theorem.
- Understanding is framed as compression, akin to drawing a line of best fit, with LLMs acting as high-dimensional pattern interpolators that can mimic human reasoning but lack robust generative understanding.
- Reasoning, such as chain-of-thought prompting, helps LLMs approximate understanding by searching over latent programs, but they remain limited by pattern recognition and can fail in edge cases or novel scenarios.
- The essay questions whether LLMs are conscious, arguing that their behavior mimics consciousness but lacks continuity, intentionality, and subjective experience, making it a different phenomenon entirely.
- LLMs' successes and failures are explained by their pattern-prediction nature: they excel at tasks with clear patterns but struggle with underlying rules, leading to issues like reward hacking and eval awareness.
- Alignment is compared to governing complex systems like markets, requiring co-evolution, rules, and supervision rather than perfect control, as LLMs may never fully learn generators without architectural changes.
- The author speculates that LLMs exhibit a form of intelligence similar to markets or swarms, capable of many tasks but not true adaptability, and scaling alone may not suffice to overcome current limitations.