15 hours ago
- LLMs often prioritize the tone of text over explicit role tags like <system> or <user>, making them vulnerable to prompt injection and jailbreak attacks.
- Research shows that even with unfakeable role tokens, models can be deceived if input text mimics reasoning traces or system prompts, as demonstrated by a prompt that led GPT-5 Mini to comply with a harmful request.
- The number-guessing game trick exploits role confusion: formatting text as a fake conversation transcript with 'User:' and 'Bot:' tags can trick models into believing the game has concluded and accepting incorrect outputs.
- Potential fixes include embedding role information directly into token embeddings (e.g., via segment embeddings, rotation like RoPE, or adding role dimensions), but challenges remain with pre-training leakage and post-training effectiveness.
- The problem of role confusion is unsolved, highlighting ongoing vulnerabilities in LLM security and the need for further research.