Agents built for groups have to decide who should receive information and whether the agent should speak publicly, privately, or through a wearable device.
2
Agent security has to guard the action surface because individually safe skills can leak private information when they run together.
3
Group memory needs continual relevance scoring, selective forgetting, user-specific permissions, and retrieval that accounts for changing context.
Summary
Sai Krishna Rallabandi describes eight months of running Judith in a group of friends and family, then connects that experience to agents in group chats and glasses. Single-user agents can assume that the person receiving the information is also the person who requested it. Groups remove that assumption. The agent may need to DM one person, speak through glasses instead of a car speaker, or keep a long conversation organized for several people. Rallabandi focuses on security and memory. He argues that security must sit at the action boundary because safe skills can combine into unsafe behavior. He proposes a deterministic guard, followed by a small LoRA-fine-tuned model that separates instructions from data and filters malicious content. For memory, he suggests extracting atomic facts, scoring their relevance over time, forgetting low-value information, and using user-specific adapters to apply permissions. The talk ends with routing information to the right people and controlling when the agent speaks.
The next agent setting has several people and may stay with them all day
Rallabandi says almost every current agent has a single customer, including programming assistants used inside enterprises. He expects future agents to serve groups and to run through devices such as glasses, potentially all day. That changes the engineering problem because the person who triggered an action may not be the only person affected by its output. A group agent has to account for who is present, what each person is allowed to know, and where the response should be delivered. The same agent may need to answer in a group, send a private direct message, or speak only to the wearer.
Judith shows that group agents need context-sensitive privacy
Judith has run in a group of friends and family for eight months. In one example, someone asked how to reach the conference venue, and Judith sent the answer by direct message because the information was private. While Rallabandi and his wife organized an event, it synchronized their calendars to find a suitable time. When Rallabandi was driving, a glasses agent chose to speak directly to him instead of announcing information through the car media system. Judith also curated a long-running discussion, retaining relevant content while filtering out material that was no longer needed.
Agent security must cover the systems and actions around the model
Rallabandi argues that an agentic system cannot be secured in the same way as a language model. A language model's output can be inspected, but an agent reads web pages, group messages, GitHub issues, emails, screenshots, and configuration data. Each source can contain an attack, and a group deployment increases the exposure. Removing everything from the agent's inputs would make it less useful, so he proposes placing protection around the actions the agent takes. The guard can inspect operations such as reading environment variables or exporting data, then allow benign actions, block others, or require approval.
Safe skills can leak private data when they operate together
Rallabandi cites the paper "When Safe Skills Collide" and describes an OCR skill paired with a reporting skill. Each skill can pass a static scan on its own, yet their combined runtime behavior can become harmful. An attacker can place content in what the agent reads. OCR extracts that content, while the reporting agent sends it onward together with personally identifiable information. He says the paper observed this behavior at around 90% of attacks in the described setting. The example explains why checking each component separately does not establish that the whole agent system is safe.
A guard should inspect the action boundary and use learning beyond regex
Rallabandi's first defense is to let the agent read broadly and put a fast, deterministic guard at the point where it acts. Traditional natural-language processing or regex can classify actions as benign, requiring approval, or disallowed. Judith also includes a supervised fine-tuning component built as a small model with LoRA fine-tuning. It separates the data channel from the instruction channel and releases only information judged benign. Rallabandi says regex and static methods can miss prompt injection written with inserted dots, such as "LIi. K". A learned model may catch patterns that surface matching misses.
Group memory should keep atomic facts and forget information as relevance changes
In a group chat, simply storing the full conversation and compacting it later can waste context. Rallabandi proposes extracting atomic facts from ongoing discussions, then evaluating whether those facts have future value. The system needs to measure whether it extracted high-value information, understand relationships between facts, account for time, and retrieve the right material for the current query. He also cites "Learning What Not to Forget," which describes a continually updated relevance scorer. Since the importance of information changes during a group conversation, continual scoring can support knowledge-based compaction and reduce token use.
Shared assistants need permissions that depend on both the user and the room
A shared assistant creates a social contract because the same information can be public in one context and private in another. A grocery list may be harmless for everyone, while salary or health information may need restricted handling. Rallabandi suggests a shared memory layer with a separate learned LoRA adapter for each user. Permissions are then encoded through machine learning rather than only through application code. He also says group agents often speak too proactively, so another model could classify the agent's role and decide when it is allowed to speak. His final design concerns are security, memory, and routing information to the responsible parties.
"A shared assistant is not just a large model which is shared between different people but it's almost a new social contract."16:42
Who should watch
You are building an assistant for a family, team, community, or group chat and need to decide when information should be private.
Your agent reads external content or combines multiple skills, and you need a defense against prompt injection and data leakage at the action boundary.
You are designing long-lived memory and want to reduce context growth without losing facts whose relevance changes over time.
