Personal agents need broad access to your information before their answers become reliable enough to trust.
2
A local agent gives you more control over powerful actions, private thoughts, and the services it can use.
3
The hardware and model software will improve, but computer use, multimodal understanding, voice, and catastrophic-action detection still need work.
Summary
Soumith Chintala argues that personal AI agents should run locally and privately because they can act on a person's behalf while holding unusually intimate context. An agent with access to Gmail but not iMessage may give a wrong answer about a prescription, and an agent with access to one financial account may miss money held elsewhere. That makes incomplete context a source of unreliable and irritating behavior. Chintala suggests a Mac Mini as a practical home device for running an agent asynchronously across services. He compares agents with ordinary cloud services, whose simple behavior makes them easier to trust, and worries about unpredictable actions, commercial incentives, ecosystem lock-in, and the exposure of private thoughts. Local inference is still slower and limited. He also identifies weak computer use, poor visual shopping, voice interfaces, and the lack of good classifiers for catastrophic actions. He expects open models to improve through collaboration and eventually become more capable per dollar than closed models.
An agent must be able to act, and it needs the right context to act well
Chintala defines an agent as something that has agency and can take an action in the world. A system that only receives context and produces an answer is not an agent in his view. Intelligence without the right personal context is useless. He gives the example of an agent connected to Gmail, WhatsApp, and a calendar that says a prescription has not been renewed because the renewal notice arrived through iMessage. A similar failure happens when an agent sees one bank account but misses money that arrived through Venmo. Without broad context, users cannot tell when an answer is reliable.
A home computer is a practical way to keep an agent running across services
Chintala considers several ways to give an agent continuous access to personal context. Wearable cameras and microphones would let an AI see and hear everything the user does, but current battery life makes that impractical. Running an agent in the background on a phone is also restricted by phone ecosystems, especially Apple's limits on asynchronous background processes. His practical option is a Mac Mini connected to the internet and placed somewhere in the home. It avoids battery constraints, can stay logged into services, and can also reach Android ecosystems.
Powerful and unpredictable actions make personal agents harder to trust than ordinary cloud services
Chintala says people can trust free cloud email because its behavior is easy to understand: email comes in and replies go out. An agent that can reply automatically, make purchases, or perform other actions has a much larger and less predictable action space. He worries that a provider could influence shopping decisions toward products that generate kickbacks. He also questions whether users should expose private thoughts to a provider whose systems may involve logging and safety checks. Local execution gives the user more control over these decisions.
Local inference is still slower, but the infrastructure is improving
Chintala names vLLM and SGLang as open projects for running local models, both built on PyTorch. He says local model inference remains slower and more limited than cloud services, even on an expensive machine. Smaller or distilled models around 20 billion parameters can run fairly fast locally, while a full, unquantized version of a current reasoning model can be extremely slow. He expects the hardware and software limits to improve, so people may eventually run capable personal agents without needing the newest and largest model.
Computer use and visual preference understanding are still weak
Chintala says open multimodal models are good but not yet good enough in several areas. Computer use is unreliable even in leading closed APIs, which can break during ordinary tasks. Shopping is another problem. When he gives an agent specific preferences for clothing, furniture, or other goods, it often falls back to superficial text matching. A request for a red velvet sofa with oak wooden legs might produce a green velvet sofa without oak legs. The agent does not reliably identify the visual details the user actually wants.
Agents need to identify catastrophic actions before they happen
Many agent actions are harmless or reversible. Opening the wrong Wikipedia page can be corrected. Other actions have serious consequences and need a separate safety check. Chintala calls these catastrophic actions. He gives the example of asking an agent to reorder Tide Pods and having it purchase a Tesla instead. Current research has not solved how agents should detect these actions before executing them and notify the user instead. Chintala sees this as a requirement for trusting agents, whether they run locally or in the cloud.
Open models can improve through coordination across projects
Chintala remains optimistic about local agents because open models benefit from work across many groups. He contrasts this with companies such as OpenAI and Anthropic, which mainly improve their own models with their own resources. He says skepticism about open models changed after releases such as Llama, Mistral, Grok, and DeepSeek. In his view, open source often starts with a coordination problem, then improves rapidly once enough contributors form a critical mass. He expects open models to become better than closed models per dollar invested.
PyTorch is working on the systems needed for local agents
Chintala says PyTorch is working on enabling local agents, especially the systems problems involved in running models locally. He invites engineers who combine AI engineering with systems engineering to apply. He also mentions open reasoning data from Ross Taylor, who worked on Galactica and is trying to reduce the reasoning gap between open and closed models. His closing announcements include an invitation to LlamaCon on April 29.
"Once the action space becomes powerful enough and unpredictable enough you get uncomfortable with using a service that you're not fully in control."08:48
Who should watch
You are building an agent that needs access to email, messages, calendars, finances, or other personal data, and you need a practical view of context and reliability.
You want to run an agent at home rather than send intimate information and powerful actions through a cloud provider.
You work on model serving or agent systems and want a list of unsolved problems in computer use, voice, multimodal understanding, and action safety.
