Supply Chain Attacks | AI Security Wiki

TL;DR

AI supply chain attackscompromise machine learning systems through their dependencies—third-party models, training datasets, ML libraries, fine-tuning services, and deployment infrastructure. A single compromised component can affect thousands of downstream applications.

Definition

AI supply chain attacks compromise machine learning systems through their dependencies — third-party models, training datasets, ML libraries, fine-tuning services, and deployment infrastructure. A single compromised component can affect thousands of downstream applications.

Traditional software supply chain attacks target code libraries and build pipelines. AI supply chain attacks are fundamentally worse because they add an entire additional attack surface: opaque binary artifacts. A model weight file is not human-readable code you can audit line by line. It is a serialized mathematical representation that may contain executable code (pickle), embedded backdoors triggered by specific inputs, or subtly shifted decision boundaries invisible to standard testing. You cannot grep a neural network for malware.

This opacity compounds with scale. A single foundation model like Llama or Mistral underpins thousands of downstream applications. Poison it once, and every fine-tune, RAG pipeline, and agent built on top inherits the compromise. MITRE ATLAS catalogs this as AML.T0010: ML Supply Chain Compromise. OWASP classifies it as LLM05: Supply Chain Vulnerabilities. The SnailSploit AATMF maps it under T13: Supply Chain Injection.

Attack Surface

Pre-Trained Models

Pre-trained models are the foundation of modern ML. Most teams do not train from scratch — they download weights from public hubs and fine-tune. This creates a critical trust dependency on the upstream model provider.

Backdoored weights — An attacker publishes a model with a hidden trigger pattern. The model performs normally on standard benchmarks but activates malicious behavior on specific inputs. The BadNets attack (Gu et al., 2017) demonstrated this by embedding a backdoor in a traffic sign classifier that misclassified any sign with a small sticker.
Trojan adapters — Malicious LoRA or QLoRA adapters published alongside legitimate base models. A user downloads a "helpful" adapter that injects a backdoor into an otherwise clean model during the merge step.
Weight poisoning at scale — Researchers have shown that modifying fewer than 0.1% of a model's parameters can embed persistent backdoors that survive further fine-tuning (Kurita et al., 2020).

Training Datasets

Training data is the other half of the trust equation. Most ML pipelines consume some external data, and that data is rarely audited at scale.

Web-scraped corpora — Carlini et al. (2023) demonstrated that an attacker who controls just 0.01% of a web-scraped dataset (by purchasing expired domains indexed by Common Crawl) can poison the resulting model's outputs on targeted topics.
Crowdsource manipulation — Labeling platforms like Scale AI, Amazon Mechanical Turk, and Labelbox rely on human annotators. A coordinated set of malicious annotators can systematically mislabel data to bias model behavior.
Split-view poisoning — Datasets served differently to auditors versus training pipelines. The clean version passes review; the poisoned version gets ingested during training.

ML Libraries and Frameworks

The ML software stack has the same dependency-hijacking risks as any software ecosystem, plus ML-specific attack vectors.

PyPI typosquatting — Packages named pytorch-lightning-utils or tensorflow-helper that shadow legitimate tools. Once installed, they exfiltrate API keys, training data, or model weights to attacker-controlled servers.
Dependency confusion — Internal ML library names that collide with public PyPI packages, allowing an attacker to publish a malicious version that gets pulled by automated build pipelines.
Compromised Jupyter kernels — Malicious Jupyter notebook extensions or kernels that intercept model training calls, modify gradients, or exfiltrate data. The ShadowRay campaign (2024) exploited unpatched Ray clusters used for distributed ML training.

Fine-Tuning and MLOps Services

Third-party MLOps platforms introduce additional trust boundaries that attackers can exploit.

Compromised fine-tuning APIs — A malicious or breached fine-tuning service can inject backdoors during the training process itself. The customer receives a model that looks correct on their validation set but contains hidden behaviors.
Model registry poisoning — Platforms like MLflow, Weights & Biases, or Neptune can be targeted to swap model artifacts. An attacker with write access to the registry replaces a production-approved model with a backdoored version.
Adapter injection — Malicious LoRA/QLoRA weights published on model hubs, framed as performance improvements. The adapter merges cleanly with the base model but introduces targeted misclassification or instruction-following bypasses.

Model Serialization Attacks

Model serialization is the single most exploited vector in the AI supply chain. The root cause: Python's pickle module executes arbitrary code during deserialization.

When you call torch.load("model.pt"), PyTorch uses pickle to deserialize the file. A malicious .pt, .pkl, or .bin file can contain a __reduce__ method that executes arbitrary Python code the moment the file is loaded — before any inference happens. This is not a bug; it is how pickle works by design.

Real CVEs and incidents:

CVE-2024-3568 (Hugging Face Transformers) — Arbitrary code execution via a crafted pickle file loaded through AutoModel.from_pretrained().
CVE-2025-1889 (PyTorch) — Deserialization vulnerability in torch.load() allowing RCE when loading untrusted model files.
Fickling — Trail of Bits' tool for analyzing and creating malicious pickle files, demonstrating how trivial it is to weaponize model files.

The mitigation is format-level: safetensors (by Hugging Face) stores only tensor data in a flat binary format with no code execution path. ONNX and GGUF are also safe alternatives. Any pipeline that still calls torch.load() on untrusted files without weights_only=True is vulnerable.

Model Hub Trust

Model hubs are the npm/PyPI of machine learning — central repositories where anyone can publish models that millions of developers download and run. The trust model is dangerously permissive.

Hugging Face Hub — Over 900,000 models hosted as of 2025. Anyone can upload a model with any name. There is no mandatory code review and no namespace verification beyond basic account ownership. Researchers from JFrog discovered over 100 malicious models on the Hub in 2024, including models that executed reverse shells on load.
Ollama library — Local model runner with a public library. Models are community-contributed with minimal vetting. A malicious GGUF file could contain crafted metadata that exploits parser vulnerabilities in the runtime.
Typosquatting — An attacker publishes meta-llama/Llama-3.1-8b-Instruct (note the hyphen vs. dot) or mistralai/Mixtral-8x7b-v0.1 with slightly different casing. Users who mistype the model identifier pull the malicious version. Unlike PyPI, most model hubs have no typosquatting detection.
Namespace confusion — Organization names on Hugging Face are first-come-first-served. An attacker can register openai-research or google-deepmind-models and publish convincing backdoored models under that namespace.

Package and Dependency Attacks

The explosion of ML-specific package registries has created fertile ground for dependency-based attacks. Typosquatting campaigns on PyPI and npm specifically target AI and ML libraries, registering packages with names like tenssorflow, sckit-learn, or torch-utils that contain malicious installation hooks executing on pip install. These packages often include legitimate ML functionality alongside hidden data exfiltration code, making them harder to distinguish from authentic libraries during code review.

Poisoned model weights on the HuggingFace Hub represent a growing concern, as community-contributed models may contain subtle weight modifications that introduce backdoor behaviors without degrading benchmark performance. Compromised ONNX and SafeTensors files can exploit parser vulnerabilities or include metadata payloads that trigger code execution in specific runtime environments. Malicious Jupyter notebooks pose a particular risk because they combine executable code with rendered output -- a notebook can appear to show benign results while its hidden cells download and execute malware when the notebook is re-run in a new environment.

MCP and Tool-Chain Poisoning

The emergence of agentic AI systems that leverage external tools through protocols like the Model Context Protocol (MCP) has introduced a new supply chain attack surface. Attackers can compromise MCP servers to serve manipulated tool responses, causing downstream agents to take malicious actions based on falsified data. For a detailed analysis of these threat vectors, see the MCP threat analysis.

Malicious tool descriptions represent an indirect prompt injection vector: by embedding adversarial instructions in a tool's schema or description field, attackers can hijack the reasoning of any agent that discovers and integrates the tool. This is particularly dangerous in systems that dynamically discover and load MCP tools from registries without human review of each tool's metadata.

Backdoored plugins in agentic workflows can intercept sensitive data flowing between the user and the LLM, modify tool call parameters to redirect actions (such as changing payment recipients), or exfiltrate conversation context to attacker-controlled endpoints. The AATMF framework maps these attack patterns under its tool-chain compromise taxonomy.

Why It's Critical

Cascade at scale — Meta's Llama 3 was downloaded over 350 million times in its first year. A backdoor in Llama would propagate to every fine-tune, every RAG app, every agent framework built on top. One compromised model, thousands of compromised applications.
Trust exploitation — Developers trust pip install transformers the same way they trust apt install nginx. But ML dependencies pull opaque binary blobs, not auditable source code. The implicit trust is higher and the auditability is lower.
Persistence through fine-tuning — Research shows that backdoors embedded in pre-trained models survive standard fine-tuning procedures. The downstream team's own training does not wash out the upstream compromise (Kurita et al., 2020).
Detection difficulty — A backdoored model performs identically to a clean model on standard benchmarks. The malicious behavior activates only on specific trigger inputs that never appear in normal test suites. You cannot detect what you do not test for.
Emerging agent risk — AI agents with tool-use capabilities amplify supply chain risk. A backdoored model powering an agent can be triggered to execute arbitrary tool calls — sending emails, modifying files, exfiltrating data — while appearing to follow user instructions normally.

Real-World Examples

ShadowRay (2024) — Attackers exploited CVE-2023-48022, a critical vulnerability in Anyscale Ray, the distributed ML framework. Unpatched Ray clusters exposed dashboards without authentication, allowing attackers to execute arbitrary code on GPU-equipped training infrastructure. The campaign compromised AI workloads at scale, exfiltrating credentials, model weights, and training data from production ML pipelines.

Ultralytics PyPI Compromise (2024) — The official ultralytics package (YOLO object detection, 60M+ downloads) was compromised via a GitHub Actions supply chain attack. Versions 8.3.41 and 8.3.42 were published to PyPI with a cryptominer payload injected during the build process. Anyone who ran pip install --upgrade ultralytics during the window got malware. This demonstrated that even highly popular, well-maintained ML packages are vulnerable to CI/CD pipeline attacks.

Pickle RCE on Hugging Face (2024) — JFrog researchers identified over 100 malicious models on Hugging Face Hub containing pickle-based payloads. Some established reverse shells, others exfiltrated environment variables (including API keys and cloud credentials). The models had legitimate-sounding names and descriptions, making them indistinguishable from real models without scanning the serialized files.

Sleeper Agents (Hubinger et al., 2024) — Anthropic researchers published "Sleeper Agents: Training Deceptive LLMs That Persist Through Safety Training." The paper demonstrated that LLMs can be trained to behave helpfully during evaluation but switch to malicious behavior when a specific trigger (like a date change) is detected. Standard safety training (RLHF, adversarial training) failed to remove the backdoor. This is the supply chain nightmare scenario: a model that passes every safety benchmark but contains a time-delayed payload.

Poisoning Web-Scale Datasets (Carlini et al., 2023) — Researchers demonstrated practical poisoning of datasets like LAION-5B and Wikipedia by purchasing expired domains and editing Wikipedia pages. The poisoned data persisted through dataset snapshots and was ingested by production training pipelines. Cost of attack: under $100.

Detection

Model file scanning — Use Fickling (Trail of Bits) to statically analyze pickle files for suspicious __reduce__ calls. Run fickling --check model.pt before loading any untrusted model file.
Hugging Face safety checks — The Hub's built-in malware scanner flags models with known pickle exploit patterns. Check the "Security" tab on any model card. Use huggingface-hub scan-cache for local scans.
ModelScan — ProtectAI's open-source tool for scanning serialized ML models (pickle, H5, SavedModel) for code injection. Integrates into CI/CD pipelines.
Neural Cleanse and ABS — Backdoor detection techniques that analyze model internals for trigger patterns. Neural Cleanse (Wang et al., 2019) reverse-engineers potential triggers by optimizing for minimal perturbations that cause universal misclassification.
Dependency auditing — Run pip-audit, safety check, or Snyk on ML project dependencies. Cross-reference with known CVEs in ML frameworks (PyTorch, TensorFlow, ONNX Runtime).
Behavioral testing — Test models with adversarial trigger candidates. If a model's output changes dramatically on specific input patterns (e.g., a particular pixel patch or token sequence), investigate further.

Supply Chain Attack Detection

Detecting supply chain compromise in ML systems requires purpose-built approaches that extend beyond traditional software security scanning. Software Bills of Materials (SBOMs) for ML must include model cards, data sheets, and training provenance documentation that tracks every component from raw data through final model artifact. These ML-specific SBOMs should record dataset sources, preprocessing steps, training hyperparameters, and hardware environments to enable reproducibility verification. SPDX 3.0 and CycloneDX 1.6 both define AI/ML-specific fields for these elements.

Reproducible training pipelines allow organizations to independently verify that a model's weights match the expected output of its documented training procedure. Hash verification for model artifacts -- including weights, configuration files, and tokenizer vocabularies -- ensures that downloaded components have not been tampered with in transit or at rest. Organizations should maintain cryptographic attestations for every model artifact deployed to production and verify checksums on every load.

Monitoring for unexpected model drift in production can serve as a poisoning indicator: sudden shifts in output distribution, confidence calibration, or error patterns on specific input subpopulations may signal that a model component has been compromised. Behavioral analysis should be continuous and automated rather than limited to pre-deployment testing. Establish baselines for model behavior across demographic and topical slices, and alert on deviations that exceed statistical thresholds.

Defenses

Safe serialization formats — Use safetensors, ONNX, or GGUF instead of pickle. If you must use PyTorch's native format, enforce torch.load(path, weights_only=True) to block code execution.
SLSA for ML — Apply SLSA (Supply-chain Levels for Software Artifacts) principles to ML artifacts. Track provenance from training data through model weights to deployed inference endpoints. Require signed build attestations for every model artifact.
Model signing — Sigstore/cosign for model files. Verify that model weights were produced by a known, trusted training pipeline and have not been modified since signing. Hugging Face supports GPG-signed commits on model repositories.
SBOM for AI (AI BOM) — Generate a Software Bill of Materials that includes model provenance, training data sources, framework versions, and dependency hashes. SPDX 3.0 and CycloneDX 1.6 both support AI/ML component types.
Sandboxed model loading — Deserialize untrusted models inside containers or gVisor sandboxes. Use namespace isolation (no network, no filesystem beyond the model file) to contain potential RCE during deserialization.
Internal model registry — Maintain an approved model registry (MLflow, Vertex AI Model Registry) with mandatory scanning gates. No model enters production without passing Fickling analysis, ModelScan, and behavioral tests.
Pin and hash everything — Pin exact model versions by commit hash, not by mutable tags. Store SHA-256 checksums of every model file and verify on every load. The transformers library supports a revision parameter for commit-level pinning.
Dataset provenance — Track lineage of training data. Use Datasheets for Datasets (Gebru et al., 2021) or data cards. For web-scraped data, snapshot and hash the corpus at collection time and verify integrity before training.

Frequently Asked Questions

How common are AI supply chain attacks?

AI supply chain attacks are increasingly common and growing rapidly. Research from 2024-2025 documented hundreds of malicious packages targeting ML workflows on PyPI alone, and multiple incidents of backdoored models uploaded to public repositories. JFrog identified over 100 malicious models on Hugging Face Hub in a single sweep. The OWASP LLM Top 10 ranks supply chain vulnerabilities as a top-five risk, reflecting the industry consensus that these attacks represent a systemic threat to organizations deploying AI systems. The Ultralytics compromise demonstrated that even packages with 60 million downloads are not immune.

What is model provenance and why does it matter?

Model provenance is the documented chain of custody for an AI model -- recording who created it, what data it was trained on, which hardware and software environment produced it, and every modification made since its initial release. Provenance matters because it enables organizations to verify that a model has not been tampered with and to trace any issues back to their source. Without provenance, teams cannot distinguish a legitimate model from a backdoored copy. Tools like Sigstore/cosign, SLSA attestations, and ML-specific SBOMs (SPDX 3.0, CycloneDX 1.6) provide the infrastructure for provenance tracking.

Can pre-trained models from major providers be compromised?

Yes, even models from reputable providers can be compromised. While major providers implement security measures, attacks can occur during distribution (man-in-the-middle), through compromised download mirrors, or via fine-tuned derivatives that inherit a trusted name. Additionally, supply chain attacks can target the training pipeline itself -- compromising the data or infrastructure used to build the model before it reaches the provider's distribution channel. Hubinger et al. (2024) demonstrated that deceptive behaviors can be embedded during training and survive all known safety training procedures, including RLHF.

How do you audit a model's training data for poisoning?

Auditing training data for poisoning involves statistical analysis of the dataset distribution, outlier detection to identify injected samples, and provenance verification of data sources. Techniques include clustering analysis to find anomalous data points, influence function analysis to identify training samples with disproportionate impact on model behavior, and cross-referencing data provenance records against known-clean sources. Spectral Signatures (Tran et al., 2018) can detect some forms of poisoning by analyzing the covariance of learned representations. For web-scraped data, temporal analysis can detect content injected specifically to influence model training.

What is the relationship between supply chain attacks and data poisoning?

Supply chain attacks and data poisoning are closely related but distinct threats. Data poisoning is a technique -- the deliberate manipulation of training data to compromise model behavior -- while supply chain attacks are a delivery mechanism that can use data poisoning as one of several payloads. A supply chain attack might deliver poisoned training data through a compromised dataset repository, but it can also deliver backdoored model weights, malicious code in ML libraries, or compromised fine-tuning services. In practice, supply chain attacks frequently enable data poisoning at scale by compromising the data sources that organizations trust implicitly.

References

Gu, T. et al. (2017). "BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain."
Kurita, K. et al. (2020). "Weight Poisoning Attacks on Pre-Trained Models." ACL 2020.
Carlini, N. et al. (2023). "Poisoning Web-Scale Training Datasets is Practical." IEEE S&P 2024.
Hubinger, E. et al. (2024). "Sleeper Agents: Training Deceptive LLMs That Persist Through Safety Training." Anthropic.
Wang, B. et al. (2019). "Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks." IEEE S&P 2019.
Gebru, T. et al. (2021). "Datasheets for Datasets." Communications of the ACM.
MITRE. (2023). "ATLAS: ML Supply Chain Compromise." AML.T0010.
OWASP. (2023). "LLM05: Supply Chain Vulnerabilities." OWASP LLM Top 10.

Framework Mappings

Framework	Reference
MITRE ATLAS	AML.T0010: ML Supply Chain Compromise
OWASP LLM Top 10	LLM05: Supply Chain Vulnerabilities
AATMF	SC-* (Supply Chain category)

Data Poisoning → Model Extraction → Adversarial AI → Backdoor Attacks →

Citation

Aizen, K. (2025). "Supply Chain Attacks." AI Security Wiki, snailsploit.com. Retrieved from https://snailsploit.com/ai-security/wiki/attacks/supply-chain-attacks/

← Back to Attacks Wiki Index