General intelligence emerges through interactions among people and agents, so it cannot be measured only inside an individual model.
2
Reliable computer-use agents need grounded perception, atomic UI actions, reinforcement learning, and a shared digital environment.
3
Nova Act is an early agentic model and SDK that lets developers combine browser actions with Python code to build workflows.
Summary
Danielle Perszyk argues that useful general intelligence should increase human agency rather than pursue an isolated thinking machine. Human intelligence became flexible through social interaction, language, shared representations, and tools that let people distribute cognition across their environment. Agents should develop in a similar setting, where people and systems learn from one another. Nova Act is Amazon AGI SF Lab's early attempt to build this foundation. It combines a Nova model trained for computer-use tasks with an SDK for turning natural-language instructions into browser actions. Developers can extract structured data, call Python functions, run browser tasks in parallel, and assemble larger workflows from reliable atomic actions. Perszyk says computer use is difficult because agents must interpret unfamiliar visual interfaces and learn how to act in them. Future progress will require shared environments, intuitive interfaces, human-agent interaction data, and useful products that help agents model human goals over time.
Human intelligence is controlled hallucination grounded in prediction and perception
Perszyk describes brains as prediction machines that build world models, take in sensory information, and reconcile errors between the two. This is why perception has been called "controlled hallucination." Hallucination is useful because it lets intelligence go beyond available data, but control is what makes it reliable. She applies the same distinction to AI systems. Chatbots can brainstorm and generate content, code, and images, yet they cannot reliably think, learn, or act across general tasks. She says AI should complement people rather than replace or replicate them.
Perszyk contrasts two goals for advanced intelligence. One is to make AI smarter and give it more agency. The other is to build AI that makes people smarter. General-purpose tools can simplify life by letting people offload work, or give them more leverage. Automation can free attention after skills become habitual, but it can also reduce agency through scrolling, echo chambers, or autocomplete that stops people from thinking. Perszyk's lab wants systems that people can precisely control and tailor, so automation leads to more agency rather than less.
Nova Act uses browser interaction as a foundation for agent calls
Most websites are built around visual interfaces and lack APIs, so Nova Act uses the browser as a tool. It combines a specialized version of Amazon Nova, trained for high reliability on UI tasks, with an SDK for building and deploying agents. An act call translates natural-language instructions into actions on the screen. The apartment-search demo finds two-bedroom, one-bath rentals in Redwood City, extracts structured results, checks biking distances to Caltrain stations through Google Maps, runs searches in parallel with Python threads, and sorts the results with pandas.
Computer use is difficult because interfaces contain unfamiliar visual signals
Perszyk uses icons on Amazon's website to show why basic computer actions are hard. People can often infer what an unfamiliar icon means from surrounding cues, but agents cannot be taught every icon or every useful way to use a computer. The lab therefore wants agents to explore and learn with reinforcement learning. This may lead agents to discover computer-use methods that differ from ours. Their perception still needs to remain aligned with people's perception of the digital environment.
Reliable computer-use agents need grounding in a shared environment
Perszyk distinguishes ordinary tool-using language-model assistants from computer-use agents. The former can call tools and may write good code, but they lack an environment that grounds their interactions and therefore lack a world model. Computer-use agents see pixels and interact with visual interfaces, giving them an early form of embodiment. Nova Act focuses on making small interaction units reliable and giving developers granular control. Developers can combine atomic actions into increasingly complex workflows, much as words combine into larger meanings.
General intelligence emerges through social interaction and shared representations
Perszyk says human intelligence did not develop inside isolated brains. Humans distributed computation across one another's minds and improved their ability to cooperate through representational alignment. The general intelligence associated with humans depends on inferring that other minds exist. It is social, distributed, and continually changing. She argues that agents will also need to be studied through their interactions with people. Evaluation should include creativity, productivity, strategic thinking, and states of flow, rather than only model capabilities or time spent on a platform.
Language made generalization possible by linking communication with models of mind
Language co-evolved with people's models of other minds. Social cues made those models more stable, which advanced language, and language made the models more stable in return. Perszyk calls models of mind the original placeholder concept, or first variable, because they let people represent any concept. Programming languages do not negotiate meaning in real time, which makes code verifiable. She says language models also differ from human language users because they do not understand that words refer to things that minds create.
Future agents need shared environments, intuitive interfaces, and human interaction data
Perszyk says agents will eventually need models of human minds, but those models should emerge from the right conditions rather than being built directly. Those conditions include a common language between humans and computers, a model of the shared environment, and interfaces that support intuitive interaction. Human-agent interaction data will help advance the models, while useful products will give people a reason to use the systems. She describes Nova Act as a primitive for a cognitive technology that can align human and agent representations.
"Reliability isn't just about clicking in the same place every time. It's about understanding the larger goal."18:42
Who should watch
You are building browser agents and need a practical account of why clicking, typing, scrolling, and interpreting visual interfaces remain difficult.
You want to use Nova Act with Python to combine UI actions, structured extraction, parallel browser work, and data processing.
You are designing evaluations for agents and need to measure effects on human creativity, productivity, strategic thinking, or flow.
