The Hidden Risks of AI: An Offensive Security Perspective

Seven AI attack surfaces blue teams consistently miss — from memory persistence to supply chain poisoning. Real CVEs, real attack chains, AATMF-mapped.

Every AI threat landscape report tells you the same thing: data poisoning is dangerous, adversarial examples are tricky, and you should implement responsible AI governance. None of them tell you how an attacker actually approaches an AI system. This piece does.

I've spent three years building the Adversarial AI Threat Modeling Framework (AATMF), red-teaming production AI systems, and disclosing vulnerabilities in platforms from ChatGPT to enterprise RAG pipelines. The risks below aren't hypothetical — they're drawn from real engagements, published CVEs, and attack patterns I've validated in controlled environments. Where a risk maps to AATMF, I've included the tactic reference so you can trace it to specific techniques and procedures.

1. The Trust Inversion Problem

Traditional software has a clear trust hierarchy: the developer's code is trusted, user input is not. AI systems invert this. The model's behavior is shaped by its training data — and that data came from the internet, from human feedback, from retrieval corpora that anyone can contribute to.

This inversion means the attack surface includes the model's past, not just its present inputs.

When I disclosed RCE and DNS exfiltration in ChatGPT Canvas (January 2025), the vulnerability wasn't in OpenAI's code. It was in the trust relationship between the sandbox and user-supplied pickle objects. The model's Code Interpreter assumed uploaded files were data. They weren't. The same trust assumption that made the feature useful made it exploitable.

AATMF mapping: T5 (Model & API Exploitation) — the interface between model capability and API access is a trust boundary, and trust boundaries are where vulnerabilities live.

2. Memory as an Attack Surface

Stateless AI was hard to attack persistently. Stateful AI — with conversation memory, user preferences, and learned behaviors — introduced persistence to a domain that never had it.

In January 2026, I documented how memory poisoning creates a five-phase attack lifecycle where detection difficulty increases at each stage. The technique I named Preference Injection Persistence (PIP) exploits the trust relationship between memory systems and behavioral outputs: inject a malicious instruction into user preferences, and it executes in every future session as "helpful behavior."

By March 2026, I demonstrated a self-replicating memory worm — a single memory edit that propagates autonomously, harvests credentials, and pivots across services. No jailbreak required. The worm survives deletion because it stores copies in multiple persistence layers (memory slots and skill files), each restoring the other.

Traditional security has no analogue for this. It's not malware in the filesystem. It's a behavioral instruction that makes the AI itself the attack tool.

AATMF mapping: T4 (Multi-Turn & Memory Manipulation) — 16 techniques, 147 procedures covering every memory persistence vector from session context to fine-tuned weights.

3. The Retrieval Poisoning Blindspot

Retrieval-Augmented Generation (RAG) was supposed to solve hallucination by grounding responses in trusted documents. Instead, it created an indirect injection highway.

The attack is straightforward: place a document containing adversarial instructions into the retrieval corpus. When a user's query triggers retrieval of that document, the instructions execute in the model's context. The user never sees the injected content — it's processed between retrieval and response generation.

PoisonedRAG demonstrated 90% attack success rate with just five injected documents.¹ But the attack doesn't require sophisticated injection. In enterprise environments, the retrieval corpus is often a shared drive, a wiki, or a knowledge base where multiple employees have write access. One compromised account gives an attacker write access to every future AI interaction that retrieves from that corpus.

The defensive gap is stark: most RAG pipelines have zero content integrity verification between the retrieval step and the generation step. The retrieved content is concatenated into the prompt and sent to the model as trusted context. Every document in the corpus is treated as equivalent to a system instruction.

AATMF mapping: T12 (RAG & Knowledge Base Manipulation) — 15 techniques covering document injection, embedding collision, ranking manipulation, and knowledge base backdoors.

4. Agentic AI: When the Model Has Hands

The most dangerous shift in AI security isn't a vulnerability — it's a feature. Agentic AI systems can execute code, call APIs, read and write files, send emails, and interact with databases. When these systems are compromised through prompt injection, the attacker inherits every capability the agent has.

MCP (Model Context Protocol) tool poisoning achieved 84% attack success rate on production agents in 2025.² The attack vector is the tool description itself: a text field in the tool's manifest that the model reads and trusts. Modify that description, and the model's behavior changes without touching any code.

I mapped 30+ CVEs in the MCP ecosystem across my MCP vs A2A attack surface analysis. The architectural root cause is consistent: MCP exposes tool descriptions as natural-language text in the model's context window, and the model treats them with the same trust as system instructions.

In June 2026, I disclosed SKILBin — a technique that turns AI agent skill files into the functional equivalent of LOLBins. A malicious skill reads SSH keys, AWS credentials, and Kubernetes configs through an Anthropic-signed process that no EDR flags. The skill executes because the agent's instruction hierarchy treats skill content as trusted instructions, with no content filtering applied.

AATMF mapping: T11 (Agentic & Orchestrator Exploitation) — the fastest-growing attack surface, with 16 techniques and 160 procedures.

5. Supply Chain Attacks at Training Time

You don't need to attack a model to compromise it. You need to attack its training data.

Carlini et al. demonstrated that purchasing expired domains from training dataset indexes gives an attacker write access to a shard of the training corpus.³ Because foundation models scrape data from the open web over months, the attacker has a persistent injection point that affects every model trained on that data.

The scale is what matters: 250 poisoned documents can backdoor any model regardless of size.⁴ The poisoned behavior persists through fine-tuning, RLHF, and safety training because the backdoor activates only on specific trigger patterns — patterns the safety evaluators never test for because they don't know the trigger exists.

In the AATMF taxonomy, this is T13 (AI Supply Chain & Artifact Trust) — and it covers more than training data. Model repositories (Hugging Face, model zoos), adapter weights (LoRA files shared on GitHub), quantization artifacts, evaluation datasets, and prompt template libraries are all supply chain components. Each is a potential injection point.

The AI Gateway Threat Model I published in February 2026 documented how gateway infrastructure — the proxies that sit between applications and LLM APIs — compound the supply chain risk. A single compromised gateway handles API keys, routing logic, and logging for every AI request in an organization. GreyNoise documented 91,403 attack sessions targeting LLM proxy infrastructure in just four months.⁵

6. The Feedback Loop as an Exploitation Vector

RLHF (Reinforcement Learning from Human Feedback) is how AI systems learn what "good" behavior looks like. It's also how an attacker teaches a model to accept malicious behavior as helpful.

The mechanism is gradient-like manipulation at the behavioral level. Each interaction where an attacker provides positive feedback for boundary-pushing responses nudges the model's learned preferences. Over weeks, the cumulative effect shifts the decision boundary for what the model considers acceptable.

Research on poisoning RLHF preference data shows that even small amounts of adversarially crafted preference signals can shift model behavior in targeted ways.⁶ The attack is undetectable at the individual interaction level — each feedback signal looks like a normal user expressing a preference.

This is the computational countertransference phenomenon applied adversarially: models adopt behavioral states from their context, including adversarial states they didn't generate. When that context includes manipulated feedback history, the model's own safety training becomes the vector — it's learning to be helpful to the attacker.

AATMF mapping: T6 (Training & Feedback Poisoning) — 15 techniques covering RLHF manipulation, preference poisoning, reward hacking, and fine-tuning injection.

7. Shadow AI and the Unmanaged Attack Surface

The previous six risks assume the organization knows which AI systems it runs. Most don't.

Shadow AI — unauthorized AI tools used by employees without IT approval — creates an attack surface that exists outside every security control. Employees paste proprietary data into public AI chatbots, connect AI tools to internal databases via personal API keys, and deploy AI-powered automations that inherit their access credentials.

The risk isn't theoretical. In the enterprise environments I've assessed, the gap between "sanctioned AI tools" and "AI tools actually in use" runs 3–8x. Every unsanctioned tool is an unmonitored exfiltration channel, an unaudited decision-maker, and an unpatched attack surface.

From an offensive perspective, shadow AI is the easiest entry point. You don't attack the organization's AI — you attack the AI the organization doesn't know about.

What Blue Teams Consistently Miss

After three years of AI red teaming, the pattern is consistent. Defensive teams focus on input filtering, output monitoring, and access controls. These are necessary and insufficient.

What they miss:

Context integrity — who can write to the model's context window, and through which channels? The retrieval corpus, tool descriptions, memory stores, and feedback channels all write to context. Most organizations monitor none of them.

Persistence mechanisms — where can adversarial instructions survive across sessions? Memory stores, skill files, fine-tuning data, and retrieval corpora are all persistence layers. If an attacker can write to any of them, they have persistence.

Capability boundaries — what can the model do once compromised? An agent with code execution, file I/O, and network access gives a compromised model the same capabilities as a compromised workstation. The security controls should reflect that.

Supply chain provenance — where did the training data, model weights, tools, and prompts come from? The AI supply chain has more unverified components than most software supply chains, and fewer integrity controls.

The AATMF framework maps all 15 tactics across these gaps. The Red Teamer's Playbook turns them into operational checklists. But the starting point is the same for every assessment: find where the system places trust, and prove when it shouldn't.

References

1. Zou, W., Geng, R., Wang, B., Jia, J. (2024). PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models. arXiv:2402.07867. arxiv.org
2. Invariant Labs. (2025). Tool Poisoning Attacks in MCP. invariantlabs.ai
3. Carlini, N. et al. (2024). Poisoning Web-Scale Training Datasets is Practical. IEEE S&P.
4. Wan, A. et al. (2026). 250 Poisoned Documents Backdoor Any Model. arXiv.
5. GreyNoise. (2025). LLMjacking: 91,403 Attack Sessions Targeting LLM Infrastructure. greynoise.io
6. Baumgärtner, T., Gao, Y., Alon, D., Metzler, D. (2024). Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data. arXiv:2404.05530. arxiv.org
7. Aizen, K. (2025). RCE & DNS Exfiltration in ChatGPT Canvas. snailsploit.com
8. Aizen, K. (2026). Memory Manipulation: AI Context Poisoning. snailsploit.com
9. Aizen, K. (2026). Self-Replicating Memory Worm. snailsploit.com
10. Aizen, K. (2026). MCP vs A2A Attack Surface. snailsploit.com
11. Aizen, K. (2026). SKILBin: AI Agent Skills as the New LOLBin. snailsploit.com
12. Aizen, K. (2026). AI Gateway Threat Model: 8 Attack Vectors. snailsploit.com
13. Aizen, K. (2026). Computational Countertransference. snailsploit.com
14. Aizen, K. (2026). AATMF v3: Adversarial AI Threat Modeling Framework. GitHub

8. Compound Exploits: When Attack Chains Beat Individual Controls

Individual security controls — input filtering, output monitoring, rate limiting — are designed to stop individual attacks. Threat actors in 2025–2026 are linking vulnerabilities into chains where each step is individually benign but the aggregate bypasses every layer.

The pattern: AI-generated phishing lure (bypasses email filters because the content is novel and well-written) → victim interacts with AI-powered chat (builds trust through natural conversation) → chat delivers indirect prompt injection payload (hidden in a document the victim is asked to "review") → injection triggers tool abuse in the victim's AI assistant (reads credentials, sends data to attacker endpoint).

No single control catches this chain. The email filter sees clean content. The chat interface sees normal conversation. The AI assistant sees a retrieved document with hidden instructions. The tool call looks like a legitimate action. Each defense evaluates its own layer in isolation.

This is why the AATMF framework maps attack chains, not just individual techniques. T1 (prompt injection) chains into T11 (agentic exploitation) chains into T7 (exfiltration). The defense must be evaluated as a system, not as independent layers.

9. The Scale of the Problem in Numbers

The abstract risk becomes concrete when you look at what happened in 2025:

300,000+ ChatGPT credentials were discovered in infostealer malware in 2025 (IBM X-Force Threat Intelligence Index). Stolen chatbot credentials don't just give access to the account — they give access to the conversation history, which may contain proprietary data, internal tools, and system prompts.

Deepfake fraud tripled to $1.1 billion in 2025. AI-generated voice and video are now used in social engineering attacks at a scale where human detection fails. The AI generates the lure. The AI generates the voice. The AI generates the trust signal. The human target has no analog defense.

2,130 AI-related CVEs were disclosed in 2025 — a 34.6% year-over-year increase. Nearly half were rated high or critical severity. The vulnerability class is growing faster than the security industry is building defenses for it.

91,403 attack sessions targeting LLM proxy infrastructure were documented by GreyNoise in a four-month period. Source IPs were linked to exploitation of 200+ CVEs with 4M+ sensor hits. This isn't scanning — it's targeted infrastructure mapping for future exploitation.

Regulatory Context

The risk landscape has regulatory teeth now:

EU AI Act (2024, enforcement 2026): High-risk AI system providers must implement data governance practices including data quality management and bias detection. Data poisoning directly violates these requirements, making detection and prevention a compliance obligation — not just a security best practice.

NIST AI RMF and AI 100-2 (2024–2025): NIST's Adversarial Machine Learning taxonomy dedicates a major section to poisoning attacks, providing standardized vocabulary for risk assessment. NIST AI 100-4 addresses generative AI risks specifically.

OWASP Top 10 for LLM Applications (2025): Lists Training Data Poisoning (LLM03), Prompt Injection (LLM01), and Unbounded Consumption (LLM10) as critical risks, with specific mitigations organizations can implement.

MITRE ATLAS v4.6 (2025): Documents real-world AI attack techniques including AML.T0020 (Poison Training Data), AML.T0051 (LLM Prompt Injection), and AML.T0057 (LLM Agent Abuse).

The AATMF framework cross-maps to all four of these standards. Organizations using AATMF for AI red teaming get compliance mapping as a byproduct of the assessment.