# Climbing the Ladder of Abstraction Amelia Wattenberger, Adept.ai | AI Engineer Summit 2023 | 16:47 Source: https://www.youtube.com/watch?v=PAy_GHUAICw Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/climbing-the-ladder-of-abstraction Published: 2023-11-03 Tags: design, multimodal ## TL;DR - Augmentation can be built from smaller automations that remove tedious steps while leaving the larger judgment task with the user. - AI interfaces should let people move between detailed and summarized views of the same information, like zooming through levels in a map. - Adept is exploring interfaces that generate these views and preserve the ability to act at every level. ## Summary Amelia Wattenberger argues that automation and augmentation are connected. Automating small tasks such as aggregating data or generating visualizations can give people more time for analysis and judgment. She compares this with spreadsheets, which automate calculations while letting accountants focus on the numbers. She then introduces a ladder of abstraction, using maps as an example of an interface that changes its visible information as users zoom in and out. She shows how the same idea could change reading and writing by using language models to summarize a book at paragraph, section, and chapter levels. Her main product example is an exploratory Adept interface for comparing Airbnb and hotel listings. The interface extracts information relevant to a user's decision, supports comparison across many listings, and keeps actions such as booking or messaging available at each view. The talk argues for AI interfaces that help users change the level at which they work without losing control. ## Key ideas ### Augmentation can be built from smaller automations [01:41](https://www.youtube.com/watch?v=PAy_GHUAICw&t=101s) Wattenberger reframes the usual automation-versus-augmentation split. If the goal is to augment a job such as analyzing data, smaller automations can handle the work around it, such as aggregating data into a table or generating visualizations. The person can then spend more time answering the question that motivated the data collection. She compares this with spreadsheets, where calculations and updates happen automatically while the accountant focuses on understanding the financial data. Her point is that automating tedious parts does not remove the larger task. It can give the user more room to do the work that requires judgment. ### Structured interfaces make general AI tools more useful [04:11](https://www.youtube.com/watch?v=PAy_GHUAICw&t=251s) Wattenberger says calculators and chatbots are flexible general tools, but they become more useful when embedded in a structured interface. The user should still drive the task while the model handles smaller frustrating steps. She gives spreadsheet formulas as an example of this pattern. The interface contains automated calculations inside a system that people can inspect and use. She expects future products to place language-model capabilities inside the tools where work already happens, rather than asking people to complete everything through a standalone chatbot. ### Maps change the information shown at each zoom level [04:50](https://www.youtube.com/watch?v=PAy_GHUAICw&t=290s) Digital maps show the same place at several levels of detail. At the Monterey Bay Aquarium, a close view can show individual buildings, names, icons, and routes. Zooming out removes that detail and reveals streets, restaurants, and destinations. At a still wider view, highways and terrain support longer-range travel, while the widest view mainly shows the shapes of states or countries. Keeping every detail visible would make the interface hard to understand, and much of that detail would not help with the current task. Each level is designed around what the user is trying to do. ### Language models can let readers and writers zoom through a book [06:55](https://www.youtube.com/watch?v=PAy_GHUAICw&t=415s) Wattenberger demonstrates a prototype using the first five chapters of Peter Pan. At one level, each paragraph becomes a one-sentence summary, with a minimap showing the reduced amount of text. Zooming out again produces summaries of roughly ten paragraphs at a time. At the highest level, each chapter becomes one sentence, allowing five chapters to fit on a page. A writer could edit the story at that level, changing pacing or plot structure, then zoom back in to inspect the resulting text. She also describes representing a story through an arc that maps mood across the book. ### Knowledge work often moves from information to action [09:47](https://www.youtube.com/watch?v=PAy_GHUAICw&t=587s) At Adept, Wattenberger says the team is training AI to use software, read screens, and take actions like humans do. Their broader view of computer work has three recurring parts: getting information, transforming or reasoning about it, and acting on it. This gives them a way to think about zooming out on any piece of information. The interface should change the presentation of the material while still helping the user complete the next step. ### A decision-focused view can remove irrelevant page details [10:49](https://www.youtube.com/watch?v=PAy_GHUAICw&t=649s) In her hypothetical Airbnb example, Wattenberger starts with a listing page containing generic information and then narrows it to the factors that matter for her decision. The interface could add walking time to Hotel Nikko, the nearest BART station and its walking time, and a summary of reviews about Wi-Fi speed. It could select a small number of useful room photos instead of showing the listing's many vanity images. At this level, the user could still reserve the listing or message the host without returning to Airbnb. The same view could apply to hotel listings, making comparisons easier across sites. ### Higher-level views can support comparison and group actions [13:03](https://www.youtube.com/watch?v=PAy_GHUAICw&t=783s) When comparing 50 listings, individual summaries would still require too much work. Wattenberger's next view shows all listings in a table with the chosen factors, such as total price, walking time, and positive Wi-Fi reviews. A further zoom turns each listing into a circle on a scatter plot, with price on the horizontal dimension, distance from the hotel on the vertical dimension, and Wi-Fi sentiment shown by color. She spots a cluster that is cheap, close, and likely to have good Wi-Fi. From that view, she can select the cluster and message those listings about early check-in, then book the one that responds with an acceptable time. ### The ladder should be easy to traverse in both directions [15:12](https://www.youtube.com/watch?v=PAy_GHUAICw&t=912s) Wattenberger is careful not to claim that higher abstraction is always better. Different tasks need different views, and people currently keep earlier views in their heads while comparing information. Her proposal is to use AI to generate several levels, connect them, and make movement between them easy. The user should be able to zoom out to compare or reason about structure, then zoom back in to inspect details. Actions should remain available instead of forcing the user back into the original application. ## Notable quotes - "If our end goal is to augment tasks or jobs, we'll still need to automate parts of them." (02:47) - "What we want is something where the technology behind chatbots is embedded into the interfaces where we're still driving, but the model's automating away the smaller tasks that we find so frustrating." (04:30) - "There's really only so much information we can fit in our brains and so many pixels on a screen, and most of that detail isn't relevant for the task we're trying to do anyway." (06:33) - "I'm not trying to argue that higher levels are better. Instead, what I'm trying to argue is that we can use AI to generate these different levels, glue them together, and make it easy to move between them." (15:31) ## Tools & references mentioned - Adept.ai - Google Maps - Lotus 1-2-3 - Excel - Peter Pan - Kurt Vonnegut - The Hobbit - The Wizard of Oz - Alice in Wonderland - Airbnb - Hotel Nikko - BART - Hotels.com - Bret Victor ## Who should watch - You are designing an AI product and need a way to combine model capabilities with an interface people can control. - You are building tools for research, writing, analysis, or comparison and want users to switch between detail and summary without losing their place. - You are interested in Adept's approach to letting AI read software, transform information, and take actions inside existing workflows. ## Related talks - [Designing AI To Scale Human Thought](https://aietalks.com/talks/designing-ai-to-scale-human-thought) (Jun Yu Tan, Tusk, 12:24) - [The Intelligent Interface](https://aietalks.com/talks/the-intelligent-interface) (Samantha Whitmore & Jason Yuan, New Computer, 18:56) - [Second Order Effects of AI](https://aietalks.com/talks/second-order-effects-of-ai) (Cheng Lou, 21:46) - [The Era of Unbounded Products: Designing for Multimodal IO](https://aietalks.com/talks/the-era-of-unbounded-products-designing-for-multimodal-io) (Ben Hylak, Dawn, 20:32) - [Form factors for your new AI coworkers](https://aietalks.com/talks/form-factors-for-your-new-ai-coworkers) (Craig Wattrus, Flatfile, 15:35)