Production ML security failures usually come from forgotten settings, excessive permissions, flat networks, and exposed secrets.
2
Security controls must fit inference latency and throughput budgets, so expensive defenses should be reserved for higher-risk workloads.
3
ML systems should be secured as infrastructure, using access control, network segmentation, encryption, secret management, and monitoring.
Summary
Lovina Dmello argues that production ML security has a familiar source: infrastructure mistakes. She opens with thousands of Ray clusters exposed on the public internet because authentication was off by default. The same pattern appears in audits of production ML systems, where excessive permissions, flat networks, and exposed credentials or model files recur. Dmello also explains why research defenses often fail to reach production. Controls add latency and reduce throughput, with heavier isolation and real-time malicious-input detection carrying significant overhead. Her proposed approach starts with basic infrastructure security, then adds access control, runtime monitoring, operational practice, and compliance. A maturity model helps teams set a production baseline and choose controls according to risk and overhead. The talk closes with newer concerns such as prompt injection, retrieval poisoning, GPU side channels, and model supply chains, while keeping its main advice practical: secure the surrounding system with the techniques already used for databases and other infrastructure.
Most production ML breaches begin with infrastructure settings
Dmello says production ML security failures are usually ordinary infrastructure mistakes rather than attacks on neural networks. She describes a 2023 discovery of thousands of Ray clusters exposed on the public internet, with dashboards and job APIs available because authentication was off by default. The exposure involved more than a billion dollars of data. She also names familiar causes behind other breaches: exposed API keys, over-privileged accounts, and model weights stored in public buckets. Her point is that a forgotten configuration setting can create the incident that reaches the news.
LLM systems changed the security assumptions inherited from older software
Dmello contrasts a classic 2008 application with a modern ML system. Older applications had deterministic behavior, fixed parameters, and a clear security boundary around one tenant. ML systems produce probabilistic behavior, protect copyable model weights, and can leak information through the API they expose. Cost pressure puts multiple tenants on shared GPUs. In LLM stacks, prompts influence control flow, retrieval corpora provide answers, and agents take actions. The infrastructure changed substantially, but teams kept many of the old security assumptions.
Infrastructure security is the foundation for the rest of the defense stack
Dmello presents four layers of defense in depth. Infrastructure security covers containers, networks, GPU virtualization, and encryption. Access control determines who can do what. Runtime security watches live system traffic. Operational practice includes team processes, compliance, incident response, and culture. Failures at the infrastructure layer can spread upward. A precise access-control policy does little if the underlying cluster is open. She also groups threats into model attacks, data and integrity problems, and classic infrastructure compromise, with real breaches often landing in the last category.
Misconfiguration is more common than missing security features
Dmello cites an audit of 50 real production ML setups in which 78% contained at least one critical security mistake. Three problems appeared repeatedly. Accounts had more permissions than they needed, allowing almost any account to do almost anything. System components were not separated, so access to one part could lead to access across the whole system. Passwords and trained models were stored where anyone could reach them. Dmello attributes the pattern partly to a division of responsibility: security teams build protections, while ML teams focus on model accuracy and may leave platform settings switched off.
Security controls have to fit the latency and throughput budget
Dmello groups controls by their production cost. Basic protections such as logins and input checks cost under about 8% in her presentation and should always be enabled. Heavier workload isolation costs 10 to 20%, so teams can apply it selectively to sensitive or untrusted workloads. Real-time malicious-input detection can cost 15 to 30%, which makes it difficult to run on every request. The practical question is how to implement a control without slowing the whole system. Cheap protections can run everywhere, while expensive ones can be reserved for systems with greater risk.
ML security tooling and team responsibilities have not caught up
Dmello says ML security tooling is years behind ordinary software security. Automated scanning and password management are established elsewhere, but comparable coverage has not fully carried over to ML. She also describes gaps between security, ML, and operations teams. Security teams may not understand ML, ML teams may not understand security, and operations teams may not know how a model behaves. Research has its own mismatch with production: papers may focus on small adversarial inputs or one model, while deployed systems face stolen passwords, misuse of existing access, and many models sharing infrastructure.
A maturity model connects protections to risk and overhead
Dmello proposes a four-level maturity model mapped to the NIST AI Risk Management Framework. Level one allows under 5% overhead and covers the bare basics, which she considers suitable for testing rather than production. Level two allows 5 to 10% overhead and includes logins, encryption, network separation, and basic monitoring. She describes this as the minimum production baseline. Level three adds advanced controls for regulated settings such as health care and finance. She says teams often believe they are at level three when their actual setup is closer to level one or two.
Basic isolation and secret hygiene address the recurring failures
Dmello gives concrete fixes for the three recurring configuration problems. Accounts should have only the permissions they need, and credentials should expire quickly. Network segments should be separated, with services required to establish a verified identity rather than relying on network access alone. Passwords should not be hardcoded, and model files should not sit in open storage. A secret manager, encryption, and automatic scanning before deployment can address those exposures. Dmello calls this basic hygiene, but says it can prevent the kinds of breaches that make the news.
Newer threats remain unsettled, so the guidance should stay practical
Dmello identifies several areas where defenses are still developing. Prompt injection can make a model confuse an attacker's input with its instructions. Retrieval-augmented systems can be affected when their document store is poisoned. Shared GPUs may create side channels through which information moves between customers. Public model and add-on sources create supply-chain risks when teams do not check where downloads came from. She describes these points as a direction rather than a final rule because threats change quickly. Her closing advice stays grounded in existing infrastructure practice: restrict access, segment networks, and protect data at rest.
"The guidance team actually need isn't should I do this, it's how do I do it without slowing everything down."11:52
Who should watch
You run ML or LLM workloads in production and want a checklist for permissions, network boundaries, credentials, model storage, and monitoring.
Your security team is adding controls that threaten inference latency or throughput and needs a way to choose where the expensive checks belong.
Your organization has separate ML, security, and operations teams that do not share a complete view of the deployed system.
