Machine learning systems can be attacked through their data, APIs, inputs, model files, and underlying software.
2
Model observability, request logging, limited outputs, and careful data handling are the first defenses against abuse.
3
Multimodal LLMs make prompt injection harder because adversarial instructions can be hidden in images that are difficult to detect.
Summary
Dr. Andrew Davis describes machine learning security as a set of problems that begin before inference and continue through deployment. He covers poisoned training data, stolen models, adversarial examples, prompt injection, unsafe model files, and ordinary software vulnerabilities. His advice is practical: verify data provenance, treat public and user-supplied data with suspicion, log model usage, watch for unusual request volumes, return only the information a user needs, and isolate models from untrusted sources. Davis is especially concerned about multimodal LLMs. Text prompt injections can be filtered with classifiers, while adversarial images can alter a model's behavior without containing obvious malicious text. He argues that teams should imagine the most damaging way an attacker could use an application, then build controls around that case. The talk also gives ML teams a familiar security task: track vulnerabilities in their frameworks, scan dependencies, and patch deployed systems.
Training data needs provenance checks before it reaches a model
Davis uses ImageNet to show how old data distribution methods create poisoning risks. The dataset originally contained URLs and labels, so users had to download the images themselves. Years later, expired domains may point to different content. He describes a person who registers expired domains, which means a download can come from an unexpected source. ImageNet did not distribute checksums for each image, so Davis recommends verifying SHA-256 hashes or other provenance information whenever it is available. He also advises skepticism toward public datasets, user submissions, and malware sources such as VirusTotal, where attackers may deliberately submit misleading samples.
RAG systems inherit the trust problems of their source material
Davis says retrieval-augmented generation systems need the same skeptical treatment as other machine learning data pipelines. A source such as Wikipedia can be edited by anyone, and a system may retrieve a false statement before an edit is rolled back. He mentions Nicholas Carlini's suggestion of examining edit history and diffs over time, rather than relying only on the snapshot retrieved at one moment. The broader advice is application-specific data cleaning. Teams need to consider where retrieved content came from, how it changed, and whether it should be treated as a fact before placing it in a model context.
API behavior can reveal enough information to recreate a model
A model theft attack can look like ordinary API use. An attacker sends many inputs, collects the responses, and trains a surrogate model from those input-output pairs. Soft targets such as logits provide more information than hard labels and can let the attacker train a useful copy with fewer queries than the original model required. Davis cites work in which researchers used about $600 of OpenAI queries to generate 52,000 instruction-following demonstrations and fine-tune Meta's Llama 7B. He recommends logging requesters, learning normal usage levels, investigating unusually high volumes, and returning only the output information the application actually needs.
Adversarial examples remain practical because attack costs are low
Davis demonstrates an image attack in which carefully calculated noise changes a panda classification while leaving the image visually recognizable. He says roughly a decade of research has produced limited progress, with strong defenses often reaching only around 50 or 60 percent robustness against advanced attacks. His economic point is direct: if an attacker spends one dollar to generate an attack and it fails, spending two or three dollars may still be worthwhile if the attack produces more money. He argues that defenses would need performance closer to the 90, 99, or 99.9 percent range to have a major effect.
The threat model changes across images, latent spaces, and tabular data
Most adversarial-example research focuses on images, where every pixel can be modified within a valid numeric range. Davis describes a different approach that moves through the latent space of a variational autoencoder trained on MNIST. This can produce an input that still looks like a digit while fooling the classifier. Tabular data raises different questions because values such as phone service or having a partner cannot be changed by tiny continuous amounts. An attacker may need to change age or another realistic feature, and the possible attack depends heavily on the application and the attacker's goal.
Multimodal prompt injection can hide where text filters cannot see it
Davis describes prompt injections as adversarial examples for LLMs. Text injections can sometimes be detected with a classifier trained on examples from sources such as Hugging Face and benign text from Wikipedia. Multimodal injections are harder because an image can contain adversarial pixel changes that affect the vision component of an LLM without presenting obvious malicious text. He gives the example of an email agent that drafts or sends messages. A textual instruction to send harmful emails may be detected, while an innocuous message with a hostile image attachment may bypass the same control. Davis recommends planning for the worst use of the application and logging what happens.
Downloaded model files can execute code during loading
Davis warns that a model file may contain more than numerical parameters. Framework convenience features can preserve executable code. In Keras, Lambda functions contain Python code, which can call functions such as exec or shutil.run when the model loads. TensorFlow also has functions that read and write files. This can give a model the behavior of a malware dropper. Davis recommends checking the organization and download count for public models, scanning them for malware, and loading uncertain files in an isolated, untrusted environment. A model with one or two downloads should not be opened in an environment containing API tokens.
ML deployments still need ordinary vulnerability management
Davis closes with software vulnerabilities in ML tools and runtimes. He cites a recently published CVE for Ollama and says newer projects often accumulate security bugs while their stability receives more attention. A vulnerable server exposed to crafted requests can allow remote code execution. His advice follows standard software security practice: track CVEs affecting ML frameworks and LLM libraries, keep container images patched, and scan them with tools such as Snyk. He wishes there were a dedicated RSS feed for this group of projects, since teams need a reliable way to learn when a package such as llama.cpp or Ollama requires an upgrade.
"If you're not doing any sort of observability or logging in your platform, you're not going to know if anybody is doing anything bad."08:53
Who should watch
You operate an ML or LLM API and need to recognize model theft, unusual usage, or information leakage.
Your RAG pipeline accepts public or user-edited content and you need practical checks for poisoned data.
You download models or run multimodal agents in production and want to understand code execution, prompt injection, and image-based attacks.
