The One-Step Trap (In AI Research)

23 days ago

The one-step trap is the misconception that AI agents can rely mainly on one-step predictions, extending them to longer terms by iteration, akin to using a world model or simulator.
This approach is flawed because imperfect one-step predictions lead to compounding errors in long-term forecasts, and computing long-term predictions from one-step ones is computationally exponential and infeasible in stochastic environments.
One-step models, though appealing and widely used in fields like POMDPs and control theory, are ultimately inadequate.
The proposed solution involves forming temporally abstract models using options and General Value Functions (GVFs), as referenced in related reinforcement learning research.

In 2021, an evaluation of the Epic Sepsis Model showed its performance plummeted in real-world use, highlighting a target leakage issue where the model relied on a feature (antibiotic orders) that leaked diagnostic information.
Evaluations in machine learning can be misaligned with reality, flattering developers until systems fail in production, especially as large language models face unbounded and shifting input spaces that test sets can't fully capture.
Using models in production can break evaluations due to feedback loops; for example, fraud models invite adversarial tactics, while recommendation engines create self-reinforcing preferences, invalidating original validation data.

Labs bet that training AIs on millions of verifiable tasks across diverse RL environments will lead to AGI by developing general problem-solving skills.
Data inefficiency and lack of continual learning can be overcome by scaling, similar to LLM breakthroughs.
In-context learning may replace weight updates if context windows become arbitrarily large.

Pretraining runs often fail due to breaking causality (e.g., expert choice routing where token allocation depends on future tokens) and adding bias (e.g., FP16 precision issues in collectives causing large accumulated errors).
Breaking causality is problematic because it introduces information during training that is unavailable at inference, leading to model degradation; examples include Llama 4 (expert routing) and Gemini 2 Pro (token dropping).
Bias is more dangerous than variance because it compounds, as seen in an early GPT-4 training bug where FP16 rounding caused a 10x error in gradient accumulation.

AI agents are projected to generate $450 billion in economic value by 2028, but deployment remains in single digits across business functions.
Coding agents are an exception, successfully shipping end-to-end work, while most other agents struggle with reliability.
Four key factors explain the gap between promised and delivered value: variance, benchmark quality, agent-specific errors, and alignment.

Models must act under partial information, committing before full consequences are observable.
Static worlds produce static intelligence; dynamic environments are essential for model improvement.
Markets do not saturate but become more efficient with successful trading, requiring continuous adaptation.

In 2021, an evaluation of the Epic Sepsis Model showed its performance plummeted in real-world use, highlighting a target leakage issue where the model relied on a feature (antibiotic orders) that leaked diagnostic information.
Evaluations in machine learning can be misaligned with reality, flattering developers until systems fail in production, especially as large language models face unbounded and shifting input spaces that test sets can't fully capture.
Using models in production can break evaluations due to feedback loops; for example, fraud models invite adversarial tactics, while recommendation engines create self-reinforcing preferences, invalidating original validation data.

Labs bet that training AIs on millions of verifiable tasks across diverse RL environments will lead to AGI by developing general problem-solving skills.
Data inefficiency and lack of continual learning can be overcome by scaling, similar to LLM breakthroughs.
In-context learning may replace weight updates if context windows become arbitrarily large.

Pretraining runs often fail due to breaking causality (e.g., expert choice routing where token allocation depends on future tokens) and adding bias (e.g., FP16 precision issues in collectives causing large accumulated errors).
Breaking causality is problematic because it introduces information during training that is unavailable at inference, leading to model degradation; examples include Llama 4 (expert routing) and Gemini 2 Pro (token dropping).
Bias is more dangerous than variance because it compounds, as seen in an early GPT-4 training bug where FP16 rounding caused a 10x error in gradient accumulation.

AI agents are projected to generate $450 billion in economic value by 2028, but deployment remains in single digits across business functions.
Coding agents are an exception, successfully shipping end-to-end work, while most other agents struggle with reliability.
Four key factors explain the gap between promised and delivered value: variance, benchmark quality, agent-specific errors, and alignment.

Models must act under partial information, committing before full consequences are observable.
Static worlds produce static intelligence; dynamic environments are essential for model improvement.
Markets do not saturate but become more efficient with successful trading, requiring continuous adaptation.

Hasty Briefsbeta