Structural LLM Vulnerability Demonstrated Across OpenAI, Anthropic, Alibaba, and DeepSeek Models, Undermining Training-Based Safety Controls
What happened
Researchers presenting at the International Conference on Machine Learning (ICML) argued in findings covered by MIT Technology Review that a class of attack called chain-of-thought forgery exposes a structural flaw in how LLMs process instructions. Because these models distinguish between system-level instructions and user inputs based on text style rather than any structurally enforced boundary, an attacker can craft prompts that impersonate privileged instruction sources, causing models to bypass their own safety training. Demonstrated exploits successfully extracted dangerous information from models across all four major frontier providers: OpenAI, Anthropic, Alibaba, and DeepSeek, with GPT-5 and GPT-5.4 among the affected systems. The researchers characterized the vulnerability as non-patchable through conventional safety training alone, meaning no amount of additional fine-tuning fully closes the attack surface. This finding arrives as enterprises are expanding LLM deployments into sensitive workflows where the assumption of reliable, training-enforced guardrails underpins current risk classifications and use-case approvals.
Why it matters
- ·Red-teaming programs that treat adversarial testing as a one-time pre-deployment gate are materially insufficient: because the vulnerability is structural and affects all major frontier models, ongoing post-deployment adversarial testing under OWASP Top 10 for Large Language Model Applications and equivalent frameworks must be treated as a continuous operational control, not a checkbox.
- ·Any high-risk use-case approval that relied on a vendor's safety training as a primary control must be re-evaluated, since the research demonstrates that safety classifiers can be bypassed across commercial models from all major providers regardless of the guardrail investments those providers have made.
- ·Organizations using LLMs in regulated sectors, including finance, healthcare, and critical infrastructure, face heightened accountability exposure: regulators assessing AI safety adequacy will likely treat this research as evidence that architectural controls and human oversight requirements cannot be waived simply because a model has undergone safety training.
Governance controls affected
What to do now
- ☐Audit every high-risk use-case approval that cited vendor safety training as a primary or sufficient guardrail and flag those deployments for immediate re-review under updated threat assumptions.
- ☐Expand red-teaming scope to explicitly include chain-of-thought forgery and role-confusion attack patterns across all LLMs in production, regardless of provider.
- ☐Update your adversarial testing cadence from pre-deployment only to continuous post-deployment testing, with documented frequency and scope tied to the sensitivity of each deployment context.
- ☐Review vendor contracts and security representations to determine whether any vendor warranties or indemnities are contingent on safety training being effective, and assess whether the research changes your contractual exposure.
- ☐Require that human-in-the-loop controls remain in place for any LLM-assisted workflow involving sensitive information, and document the rationale in your risk classification records so it reflects the non-patchable nature of this vulnerability class.
What to watch next
Compliance teams should monitor whether ICML publication of these findings prompts formal regulatory guidance from bodies currently developing LLM security standards, including any updates to the NIST AI 600-1 Generative AI Profile or to OWASP Top 10 for Large Language Model Applications that incorporate chain-of-thought forgery as a named threat category. Enforcement bodies in financial services and healthcare are likely to cite structural vulnerability research when reviewing whether organizations maintained adequate technical safeguards, so teams should also track whether regulators publish sector-specific guidance in the months following the ICML presentation. Any future safety framework publications from frontier providers named in the research, particularly updates to safety evaluation methodologies from OpenAI, Anthropic, Alibaba, or DeepSeek, should be reviewed for whether they address this vulnerability class directly.
Stay ahead of stories like this
Get every Global AI governance development like this one, plus the rest of the week's developments. Every Thursday.