Computers should combine explicit inputs, such as typing and speech, with context such as location, posture, tone, and emotion.
2
Interfaces for generative AI can use familiar physical metaphors, including light, fluid materials, fog, and blending, instead of rigid document metaphors.
3
A reasoning layer can choose both what an application says and how it presents that response across voice, gesture, projection, and visual UI.
Summary
Samantha Whitmore and Jason Yuan argue that AI applications should adapt to people and their surroundings instead of making people adapt to fixed controls. They begin with a working demo that uses pose detection to decide whether a computer should listen for speech or use keyboard and text interaction. They then show speculative interfaces for near-term hardware: social gestures for handling calls, an iPad canvas where generated images behave like physical materials, and a mixed-reality system that combines voice, gestures, projected UI, and objects in the user's environment. Their design approach treats every available signal as possible input. An LLM can reason over those signals, update its understanding of the user's situation, and choose an appropriate output modality. They are also honest about the limits. Probabilistic interfaces can produce confusing results, so users need graceful ways to correct mistakes, while social norms and context should constrain what an AI does.
Computers can be redesigned from first principles around intelligence
Whitmore and Yuan say New Computer began with the belief that people have taken familiar computing metaphors, abstractions, and tools for granted for decades. They want to reconsider how people relate to computing and intelligence. They describe intelligence as taking in information from different sources and volumes, making sense of it, reasoning about meaning, and choosing a reaction. Humans follow a loop of sensing, thinking, and reacting. ChatGPT showed them an early version of a computer approximating that loop through language input, reasoning, and language output. Their demos explore ways to make interaction feel more natural than a language-only exchange.
A computer can choose its input and output from physical context
The first demo uses a camera to measure Whitmore's pose relative to the screen. When she is close to the keyboard and facing the display, an LLM reasons that the computer should use keyboard input and text output. When she walks away, the system can listen for speech and answer aloud. The point is to avoid making someone explicitly choose between typing and talking. Whitmore and Yuan suggest combining explicit signals, such as typed words, with implicit signals, such as position, tone, and emotion. The system can use those signals to update its world state and decide how to respond.
The speakers argue that people already respond to one another through context, attention, posture, and social cues. Computers should also adapt to where a person is and what they appear to be doing. Their demo is described as a simple, hacky system that could be built with current hardware. A computer near the keyboard can focus on text interaction, while a person moving around the room can have a spoken conversation with it. The LLM is used as a reasoning layer over the sensed context. The speakers also suggest that an agent could use information about other people in the room, while taking care over when that information should be brought into a conversation.
Social gestures can express intent without navigating controls
Yuan imagines wearable hardware with sensors for hand tracking and eye tracking. In his example, a person declines a call from an ex through a gesture and expression rather than finding a hang-up button or speaking a command. The system could interpret explicit gestures, as well as speed, mood, tone, and distance. He warns that gestures carry different meanings across cultures, so developers need to consider local social norms as applications and hardware reach different places. The interface is meant to understand a person's intent from the situation and the gesture together.
Generative content can use physical metaphors such as light and blending
In a speculative iPad demo, Yuan imagines a canvas for Midjourney where generated images can be moved, combined, and remixed. He creates a Balenciaga Naruto image, asks an AI for pictures of his cat, and blends the images together. Selected images glow as they are picked up, using light as a metaphor for concentrating information into a new form. He then imagines shaking a generated result like a drink to remix it. His broader point is that generative intelligence is fluid and probabilistic, so metaphors such as fog, mercury, squishy materials, light, and metaballs may fit better than rigid metaphors based on wood, paper, or metal.
Mixed-modal interfaces can connect digital actions to the physical world
Yuan's final demo imagines a wearable with a camera, microphone, and projection. While looking through a book, the user can point at an image and say that it is interesting. The system acknowledges the selection through projected feedback. A mistaken selection can be flicked away like dust instead of being corrected with an undo button or command. The user can also ask the AI to identify a friend associated with a Halloween plan. The projected interface is tied to the objects and context already around the user, so the person does not need to retrieve a phone or issue a long command.
The speakers say voice input does not require voice output, gesture input does not require gesture output, and visual input does not require visual output. An application can combine typing speed, tone of voice, camera data, gestures, and other signals, then pass them to an LLM. The reasoning layer can decide both what to say and how to present it. Outputs might appear as words, sound, projected elements, or other forms. They also caution against using every modality everywhere. Generating a photo on an Apple Watch is an example of an interaction that may be possible but still inappropriate for the context.
Probabilistic interfaces need familiar correction paths and social limits
Because probabilistic interfaces can produce many different outcomes, the speakers recommend grounding them in familiar metaphors from nature, physics, human-made materials, and existing controls such as buttons. Users need a graceful way out when the system misunderstands a gesture or selection. Social norms also limit what an agent should do. They give the example that a banking AI agent should not suddenly move into a deep philosophical conversation because that does not fit the social setting. They leave open the question of what a truly intelligent interface will look like beyond current human-centered abstractions.
"For the first time what feels like 40 years we can finally change all of that and we can start thinking from first principles what our relationship not only with Computing but with intelligence period should look like in the future."00:34
Who should watch
You are building an AI application and need to decide when users should type, speak, gesture, or receive visual feedback.
You are designing for cameras, microphones, wearables, or mixed reality and want examples of how context can influence interaction.
You are interested in generative interfaces and need practical ideas for correction, physical metaphors, and social boundaries.
