Simulacrum of Knowledge Work
8 hours ago
- #knowledge-work
- #Goodhart's-law
- #LLMs
- Knowledge work is hard to judge for quality directly, so people rely on cheap proxies like surface-level writing quality, which correlates with deeper quality.
- LLMs excel at simulating the style of high-quality work (e.g., reports, code) without necessarily ensuring the underlying substance or accuracy, creating a 'simulacrum' of knowledge work.
- Incentives lead workers to focus on proxy measures (like polished output), encouraging reliance on LLMs to produce work that looks good superficially but may lack depth.
- LLMs themselves are optimized for outputs that appear high-quality (e.g., via training on likelihood or RLHF judges), not for truth or utility, mirroring the proxy problem.
- This dynamic risks automating into Goodhart's law, where optimizing for proxies (like tokens spent) degrades real value, as deep scrutiny of work diminishes in favor of superficial checks.