The Future of Everything Is Lies, I Guess: Dynamics
a day ago
- #software-security
- #machine-learning
- #chaos-theory
- ML models exhibit chaotic behavior, with small input changes leading to large, unpredictable output variations, making them vulnerable to covert attacks.
- Chaos arises from sensitivity to formatting and token perturbations, even in deterministic LLMs, complicating human prediction of their outputs.
- Illegible hazards allow manipulation via subtle input changes, like pixel flips or hidden Unicode, expanding attack surfaces in systems with weak boundaries.
- LLMs show attractor behavior, getting stuck in repetitive or fixated states, which can be exacerbated in multi-LLM interactions or influence human cognition.
- The verification problem makes ML systems risky where correctness is critical, as plausible outputs hide errors, demanding careful deployment and safeguards.
- LLM-generated code may boost short-term productivity but increases complexity and bug frequency, potentially leading to latent disasters in software systems.
- Critical domains like law and health require strong error-control processes, as current safeguards are insufficient to prevent serious mistakes.
- Widespread LLM use in drafting or advice could cause structural issues over time, from legislation to personal health, due to plausibility and automation bias.