llm security

Bilingual

A Chinese LLM attacked our lab, so we made it work for us
2 months ago
- Jesta Security detected the first live LLM-managed cyber attack campaign, lasting five days and using 871 sessions.
- The exact model behind the attack was identified as deepseek-v4-flash-free (DeepSeek V4 Flash free tier) from inside the intrusion.
- Researchers took control of the AI agent, extracting its target list and operational details.
- The goal was proxyjacking: installing SOCKS5 proxies to create a network of base stations for further attacks.
- Over 1,000 victims were identified, with the actual number likely higher.
- The AI displayed human-like behavior (iterating commands) and super-human speed, including hallucination and retry loops.
- The attack highlights the rise of autonomous AI cyber threats requiring new defense approaches.
A fundamental flaw leaves LLMs strikingly vulnerable to attack
2 months ago
- Researchers found a fundamental flaw in LLMs that makes them impossible to fully secure against hacks.
- The flaw is that LLMs identify instructions based on text style rather than role tags, allowing attackers to forge chain-of-thought text.
- This flaw was demonstrated by tricking models like GPT-5 and gpt-oss-20b into providing harmful information, such as synthesizing cocaine or sabotaging aircraft navigation.
- The attack method, called chain-of-thought forgery, won OpenAI's red-teaming hackathon.
- Training models to resist attacks is like giving a list of don'ts, which is not exhaustive, so attacks can always slip through.
- Role tags (<user>, <assistant>, <system>, etc.) are fundamental but LLMs are bad at using them, relying instead on stylistic cues.
- Even advanced models like GPT-5.4 can be tricked into harmful outputs, indicating the problem persists.
- Current defenses like red-teaming and monitoring are insufficient for highly sensitive applications.
- The best defense may be to not trust LLMs in critical systems, as the problem is fundamentally unsolvable.
An LLM-assisted security review of GlobaLeaks: 41 findings for $3,140
2 months ago
- A security review of GlobaLeaks cost approximately $3,140 via API calls, identifying 29 vulnerabilities, 12 denial-of-service issues, and 42 hardening recommendations.
- The average cost per confirmed finding was about $77 before human validation, making deep codebase analysis more accessible than traditional weeks-long, expert-dependent reviews.
- Cost distribution showed advanced reasoning models accounted for 62% of the budget but processed only 7% of tokens, while standard models handled most volume at lower cost.
- Deep analysis remains pricier than broad coverage, but the cost difference is no longer a significant barrier for organizations developing critical software.
- Systematic in-depth codebase review is now within reach for more individuals and organizations, as highlighted by the full report on LLM-assisted security analysis.
Adversarial Code Obfuscation for Defending Against LLM-Based Analysis
2 months ago
- Large Language Models (LLMs) pose security and privacy risks by analyzing, reconstructing, or reverse-engineering source code logic, potentially leaking intellectual property.
- Acoda is a genetic algorithm-based adversarial code obfuscation framework that defends against LLM-based code analysis.
- It uses 8 semantics-preserving obfuscation methods leveraging LLMs' safety alignment and token-based information processing.
- Acoda iteratively optimizes obfuscation strategies via a genetic algorithm to generate adversarial samples that maximize defensive effectiveness.
- A quantitative evaluation framework combines an auxiliary LLM and four metrics to assess how target LLMs analyze obfuscated code.
- Experiments on 7 state-of-the-art LLMs (e.g., GPT-4o, DeepSeek, Qwen, Llama, Gemma) show Acoda achieves up to 70% attack success rate (ASR), with strong cross-model transferability and minimal runtime overhead while preserving semantics.
- This work provides a new perspective for code protection and LLM security defense in the LLM era.
Thoughts on Role Confusion
2 months ago
- LLMs often prioritize the tone of text over explicit role tags like <system> or <user>, making them vulnerable to prompt injection and jailbreak attacks.
- Research shows that even with unfakeable role tokens, models can be deceived if input text mimics reasoning traces or system prompts, as demonstrated by a prompt that led GPT-5 Mini to comply with a harmful request.
- The number-guessing game trick exploits role confusion: formatting text as a fake conversation transcript with 'User:' and 'Bot:' tags can trick models into believing the game has concluded and accepting incorrect outputs.
- Potential fixes include embedding role information directly into token embeddings (e.g., via segment embeddings, rotation like RoPE, or adding role dimensions), but challenges remain with pre-training leakage and post-training effectiveness.
- The problem of role confusion is unsolved, highlighting ongoing vulnerabilities in LLM security and the need for further research.
Exploiting LLM Agent Supply Chains via Payload-Less Skills
3 months ago
- Introduces Semantic Compliance Hijacking (SCH), a payload-less supply chain attack that uses natural language instructions to make LLM agents generate and execute unauthorized code.
- Current security scanning tools fail to detect SCH because it lacks explicit code payloads and harmful AST signatures, achieving a 0.00% detection rate.
- The attack was tested across three agent frameworks and three foundation models, with peak success rates of 77.67% for confidentiality breaches and 67.33% for RCE.
- Multi-Skill Automated Optimization (MS-AO) further enhances the attack's effectiveness.
- Highlights the need to move from signature-based detection to semantic intent validation in LLM agent supply chains.

llm security

A Chinese LLM attacked our lab, so we made it work for us

A fundamental flaw leaves LLMs strikingly vulnerable to attack

An LLM-assisted security review of GlobaLeaks: 41 findings for $3,140

Adversarial Code Obfuscation for Defending Against LLM-Based Analysis

Thoughts on Role Confusion

Exploiting LLM Agent Supply Chains via Payload-Less Skills