Machine Learning Security: Key Threats, Attacks, and Defenses

Machine learning security is no longer a research boundary problem. As ML models move into fraud detection, malware classification, autonomous threat response, and access control, they inherit a threat surface that traditional application security was not designed to handle. Attackers are not just targeting the applications that run on ML systems; they are targeting the models themselves.

NIST formalized this landscape in March 2025 with the release of AI 100-2e2025, an updated taxonomy covering adversarial machine learning attacks and mitigations across both predictive and generative AI systems. The document establishes common terminology across evasion, poisoning, privacy, and misuse attack categories, a signal that the field has matured enough to need shared vocabulary for compliance and governance purposes. The generative half of that surface, where the same failure modes appear as prompt injection and excessive agency, is catalogued separately in the OWASP LLM Top 10 breakdown, and both sit inside the AI security threat intelligence hub.

The harder gap, though, is governance rather than technology. Adversarial testing tools exist. The problem is that most security teams do not have an inventory of which models are in production, where those models came from, or what the chain of custody looks like for the weight files being served at inference.

The Core Attack Types

Data poisoning is the attack most organizations are least prepared to detect. An adversary injects malicious samples into training data before or during the model training cycle, causing the resulting model to learn incorrect patterns. In a spam filter, poisoned training data can carve out specific senders as perpetually clean. In a network intrusion detection system, it can train the model to ignore particular traffic signatures. The attack happens before the model is deployed, which means it produces no runtime anomalies that monitoring tools can catch.

Evasion attacks hit deployed models at inference time. The attacker modifies input data, often in ways imperceptible to humans, to cause a misclassification. In image recognition, the classic example is adversarial perturbations: pixel-level changes that flip a model’s prediction with high confidence. In network security, the equivalent is packet manipulation, where carefully adjusted timing, size, or encoding makes malicious traffic appear normal to an ML-based detector. ISACA’s 2025 analysis of adversarial ML in cybersecurity contexts notes that evasion is the most operationally immediate threat to ML-based security tooling. The attacker needs only query access to the model, which is available in almost every deployed system.

Model extraction, which the UK NCSC calls model characterisation, allows an adversary to replicate a proprietary model by querying it systematically and training a surrogate on the outputs. The surrogate does not need to be identical; it needs to be close enough to probe for weaknesses. In practice, extracted models are used to develop evasion inputs offline, where there are no rate limits or detection mechanisms, then replay those inputs against the original system. This two-stage pattern is increasingly common against production fraud detection APIs.

Membership inference and model inversion are privacy-layer attacks rather than reliability attacks. Membership inference determines whether a specific data record was part of a model’s training set, exploiting the tendency of models to behave with subtly higher confidence on training samples than on novel inputs. Model inversion goes further and reconstructs training data from the model’s own outputs. For healthcare, financial, or HR applications trained on real user data, both leak information about individuals without the attacker ever breaching the underlying data store.

Backdoor attacks embed hidden triggers during training that cause specific behavior, often misclassification, when that trigger appears in a future input. A facial recognition system with a backdoor might grant access to anyone wearing a specific pattern. The model behaves normally on all other inputs; the trigger activates only when the attacker chooses. Detecting backdoors requires access to training data provenance and dedicated testing with synthetic trigger patterns, neither of which is standard practice in most MLOps pipelines.

The UK National Cyber Security Centre’s taxonomy adds two more classes that map to the training environment rather than the model: malicious model training and model artifact manipulation, both of which fall under supply chain risk. The NCSC’s seventh category, hardware side-channel attacks, remains an active research problem with limited production-ready mitigations available today.

The Model Supply Chain Gap

Training data poisoning and model artifact manipulation converge on a shared structural problem: the model supply chain is largely unverified in most enterprise deployments.

OWASP’s LLM Top 10:2025 classifies supply chain compromise as LLM03 and cites several concrete attack vectors:

Pre-trained model tampering. Direct parameter editing, demonstrated publicly by the PoisonGPT proof-of-concept, modifies model behavior in ways that survive standard benchmark evaluation but activate only under specific inputs.
LoRA adapter injection. Malicious low-rank adaptation adapters merged into base models can introduce backdoors that are not visible in base model weights and that survive the merge process.
Weak provenance. Most publicly distributed model weights carry no cryptographic signature. A model downloaded through unofficial redistribution channels, and quantized GPTQ or GGUF formats are frequently redistributed outside original repositories, may not match what it claims to be.

OWASP’s recommended controls include signed software bills of materials (SBOMs) for model artifacts, red-team evaluation of third-party models before integration, and anomaly detection on model outputs in production. None of these require novel capabilities, but most organizations have no process to apply them systematically.

ML Pipeline Security: Where Vulnerabilities Accumulate

The attack surface for machine learning systems extends well beyond the model itself. The Cloud Security Alliance’s 2025 analysis of ML pipeline threats identifies five distinct stages where security controls are typically weak: data ingestion, training infrastructure, model storage, deployment APIs, and monitoring systems.

Supply chain exposure is particularly acute at the dependency layer. Modern ML pipelines depend on pre-trained foundation models, open-source libraries, third-party datasets, and hosted inference endpoints. A single compromised dependency, whether a tampered model checkpoint from a public registry, a poisoned dataset in a shared storage bucket, or a malicious package update in a training framework, can propagate corruption across every model trained on that infrastructure.

Access control fragmentation compounds the problem. A typical MLOps environment spans data warehouses, training clusters, model registries, feature stores, and serving infrastructure, often with different identity systems and permission models for each. Misconfigurations in any layer create paths for lateral movement that would not exist in a more unified application stack.

Monitoring gaps are the default, not the exception. Traditional security tooling monitors application behavior at the network and endpoint layers. It does not monitor model prediction distributions, feature drift, or decision boundary shifts, which are the signals that would indicate an active evasion campaign or a poisoning event taking hold. Teams without ML-specific observability have no baseline from which to detect that something has changed. Sentryml.com covers the MLOps tooling and monitoring practices that address this gap, and defensive guardrail tooling for content filtering and output safety is covered at GuardML.

NIST AI RMF: The Governance Anchor

NIST’s AI Risk Management Framework, released January 2023, organizes ML security controls under four functions: Govern, Map, Measure, and Manage. Govern addresses organizational accountability, meaning who owns AI risk, what policies exist, and how risk decisions are escalated. Map identifies specific AI risks present in a given system. Measure assesses those risks. Manage defines how identified risks are treated and tracked.

The framework’s most practical application is as an audit structure: walk through each function and identify gaps between what the framework expects and what the organization can demonstrate. A companion Generative AI Profile, published in July 2024, extends coverage to generative-system-specific risks including prompt injection, model abuse, and generated content harms, which are mapped against the OWASP and Cisco frameworks in the practitioner guide to generative AI risks.

The NIST AI 100-2e2025 taxonomy is the current authoritative reference for the technical layer. It is voluntary guidance, not a compliance requirement, but it provides the shared terminology that makes risk discussions across security, data science, and legal functions tractable.

Defender Action Items

Adversarial testing before deployment is the most direct technical control available. Red teams with ML expertise can generate evasion inputs, test model behavior under distribution shift, and probe for trigger-based backdoors using published attack libraries. This is not a one-time exercise; models retrained on updated data inherit new vulnerabilities that require retesting.

Beyond that, five controls address the highest-likelihood risks:

Build a model inventory. List every ML model in production with source, hash, and deployment date. Until this list exists, scope is unknown and governance is impossible.
Verify model provenance before integration. Treat unverified third-party model weights as untrusted binaries. Require cryptographic verification or internal red-team evaluation before production promotion.
Instrument model outputs. Establish baseline output distributions at deployment and alert on significant statistical deviation. Poisoned models frequently produce detectable anomalies when triggered.
Harden the training pipeline. Access controls on training datasets, logging of data modification events, data provenance records covering the origin, timestamp, and hash of every training sample, and change management for hyperparameters are baseline controls that most training infrastructure does not enforce by default.
Apply the NIST AI RMF Govern function first. Without organizational accountability for ML risk, technical controls do not have a home. Designate ownership before purchasing tooling.

Input sanitization at the serving layer, through feature squeezing, input transformation, and anomaly scoring on incoming requests, reduces the attack surface for evasion. No sanitization scheme is complete, but raising the cost of successful evasion meaningfully reduces opportunistic attacks. Strict access control across the entire ML pipeline, with unified identity, least-privilege service accounts, and separation between training and serving environments, limits blast radius if any component is compromised.

Mlcves.com maintains a database of documented ML vulnerabilities and CVEs, a useful reference when assessing which attack patterns have been exploited against real deployed systems. AI-Alert.org maintains a running tracker of disclosed ML vulnerabilities and model compromise incidents as they are publicly reported.