23 days ago
- The one-step trap is the misconception that AI agents can rely mainly on one-step predictions, extending them to longer terms by iteration, akin to using a world model or simulator.
- This approach is flawed because imperfect one-step predictions lead to compounding errors in long-term forecasts, and computing long-term predictions from one-step ones is computationally exponential and infeasible in stochastic environments.
- One-step models, though appealing and widely used in fields like POMDPs and control theory, are ultimately inadequate.
- The proposed solution involves forming temporally abstract models using options and General Value Functions (GVFs), as referenced in related reinforcement learning research.