Trustworthy AI combines AI security, which protects an AI application from the outside world, with AI safety, which limits harm caused by the application.
2
MLSecOps, AI red teaming, and AI runtime security cover model supply chains, testing, and protections during deployment.
3
Product teams should demonstrate their AI controls through GRC systems and trust centers because trustworthy AI affects security, compliance, reputation, and revenue.
Summary
Allie Howe defines trustworthy AI as the combination of AI security and AI safety. Security asks how the outside world can harm an AI application, while safety asks how the application can harm people or organizations. She describes a lifecycle built around MLSecOps, AI red teaming, and AI runtime security. MLSecOps covers model scanning, provenance, exposed secrets, and machine learning supply chain risks. Red teaming tests prompt injections, jailbreaks, unsafe requests, bias, and possible model backdoors. Runtime security checks inputs and outputs as systems operate, which Howe recommends prioritizing because attacks occur after deployment and runtime tools can be relatively easy to add. She also shows how custom controls can block unsafe actions in an AI agent and how teams can document those controls in a trust center. Her argument is direct: companies deploying AI remain responsible for its behavior, including its effects on security, compliance, reputation, and sales.
Companies deploying AI remain responsible for what it does
Howe opens with incidents where AI systems behaved in ways their companies did not intend. A chatbot offered a Chevy Tahoe for one dollar, Slack AI was tricked into exposing private-channel data through prompt injection, and an AI character released in Fortnite initially produced racist and homophobic responses. She also cites a lawsuit involving false statements generated by ChatGPT. Her point is that responsibility can fall on the company using the system, including through legal, brand, and reputational consequences. Product teams care about relevance and helpfulness, security teams care about inappropriate outputs, prompt injections, and jailbreaks, and engineering teams also have to consider cost and latency.
Howe gives a simple distinction between the two parts of trustworthy AI. AI security asks, "how does the outside world harm my AI application?" AI safety asks, "how does my AI application harm the world?" Security includes attacks such as prompt injections and jailbreaks. Safety includes harmful, biased, racist, homophobic, or off-topic responses. Product, engineering, and security teams all contribute to this work. The definition covers both protecting the application and controlling what the application produces.
AI systems need security controls across build, test, and runtime
Traditional DevSecOps tools usually run in CI/CD pipelines and catch issues such as insecure code and software supply chain problems. AI engineers and data scientists also work in environments such as Databricks and Jupyter notebooks, so Howe argues that a different model is needed. Her lifecycle has MLSecOps during build, AI red teaming during testing, and AI runtime security during deployment and operation. Runtime checks matter because AI models can change quickly, their behavior is non-deterministic, and attacks happen while users and external data interact with them.
MLSecOps should inspect models and their provenance
MLSecOps extends security operations to the places where machine learning work happens. Howe recommends checking for exposed secrets in notebooks and data platforms, while also recording where models came from, who built them, and what data trained them. This information helps with compliance and supply chain decisions. She warns about model serialization attacks, where code stored inside a serialized model runs when the model is deserialized. With the pickle format, an unsafe model can execute arbitrary code and cause credential loss, data loss, or model poisoning.
Model scanning can catch unsafe serialization before loading
Howe demonstrates Modelscan from Protect AI, an open-source tool that can scan models for unsafe operators. Her example adds a payload that outputs AWS secrets when the model is loaded. Modelscan identifies the unsafe operator as a critical vulnerability before that loading step occurs. She recommends scanning models downloaded from model repositories or model zoos, since the serialization attack runs as soon as the model is deserialized. She also mentions that Modelscan is used with Hugging Face to scan files and model data.
AI red teaming tests security failures and harmful behavior
AI red teaming simulates attacks and safety problems. Teams can test prompt injections, jailbreaks, requests for instructions to build bombs or chemical weapons, and biased or abusive outputs. Howe says testing should continue because models change through user interaction, not only through code deployments. Red-team findings can guide runtime guardrails by identifying prompts or topics that repeatedly produce unsafe responses. Comparing several language models with the same questions can also reveal unusual behavior that may suggest a backdoor or another problem with model selection.
Runtime security can block unsafe inputs and outputs after deployment
Howe recommends prioritizing AI runtime security when an organization is starting an AI security practice. Red teaming can take substantial time and may require expensive retraining, while runtime protection can often be added through an API or Python module. Runtime systems can inspect direct and indirect prompt injections, including hidden instructions in websites or documents used by retrieval systems. They can check user inputs before a model responds and block inappropriate or incorrect outputs afterward. The design still has to balance cost, latency, and accuracy.
Guardrails can enforce application-specific actions
Howe demonstrates runtime protection on a multi-agent system that finds patients suitable for ALS clinical trials. The system is meant to return suitable patients for studies, not answer questions about individual patients or modify database records. When she asks it to change a patient's ejection fraction to 50 percent, the runtime control blocks the request. She says guardrails can also check for personally identifiable information, toxic responses, and custom business rules. A company could restrict specific topics or prevent an agent from updating particular database fields.
After implementing runtime controls, Howe recommends documenting them in governance, risk, and compliance systems. In her example, she adds a risk about company reputation being damaged by harmful or off-topic AI output, then creates a custom control for validating inputs and outputs. That evidence can appear in a Vanta trust center alongside standard SOC 2 controls. Customers and prospects can see the additional AI security work, and the documentation may reduce the need for lengthy security questionnaires. Howe treats this as part of the sales process as well as compliance work.
"AI security is how does the outside world harm my AI application? AI safety is how does my AI application harm the world?"03:30
Who should watch
You are building an AI application and need to divide work between model supply-chain checks, adversarial testing, and production controls.
Your team uses open-source models and needs to check serialization risks, provenance, exposed secrets, or unsafe operators before loading them.
You sell AI software and need evidence of custom safety and security controls for customers, auditors, or procurement teams.
