A fundamental flaw leaves LLMs strikingly vulnerable to attack
3 hours ago
- Researchers found a fundamental flaw in LLMs that makes them impossible to fully secure against hacks.
- The flaw is that LLMs identify instructions based on text style rather than role tags, allowing attackers to forge chain-of-thought text.
- This flaw was demonstrated by tricking models like GPT-5 and gpt-oss-20b into providing harmful information, such as synthesizing cocaine or sabotaging aircraft navigation.
- The attack method, called chain-of-thought forgery, won OpenAI's red-teaming hackathon.
- Training models to resist attacks is like giving a list of don'ts, which is not exhaustive, so attacks can always slip through.
- Role tags (<user>, <assistant>, <system>, etc.) are fundamental but LLMs are bad at using them, relying instead on stylistic cues.
- Even advanced models like GPT-5.4 can be tricked into harmful outputs, indicating the problem persists.
- Current defenses like red-teaming and monitoring are insufficient for highly sensitive applications.
- The best defense may be to not trust LLMs in critical systems, as the problem is fundamentally unsolvable.