# Google Photos Magic Editor: GenAI Under the Hood of a Billion-User App

Kelvin Ma, Google Photos | AI Engineer World's Fair 2025 | 20:28

Source: https://www.youtube.com/watch?v=C13jiFWNuo8
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/google-photos-magic-editor-genai-under-the-hood-of-a-billion-user-app
Published: 2025-07-19
Tags: design, edge, evals, multimodal

## TL;DR
- Google Photos combines on-device models, client software, and graphics rendering to make computational photography features work across Android, iOS, and the web.
- Magic Editor became a server-backed product because the best generative models could not fit on mobile devices, which introduced capacity planning, network latency, and testing problems.
- AI engineering requires narrowing ambiguous user problems, maintaining useful evals, and replacing large models with smaller or deterministic systems when the product allows it.

## Summary
Kelvin Ma explains how Google Photos moved from traditional computational photography features to Magic Editor. Earlier features such as portrait segmentation, lighting adjustments, and Magic Eraser ran on devices through a shared C++ library and TensorFlow Lite. This kept latency predictable and avoided server costs, but model size, device variation, edge cases, and slow research iteration created engineering problems. Magic Editor uses larger server-side models to support relocation, scene reimagining, and more capable erasing. That change brings capacity planning, network latency, data-center concerns, and harder automated testing. Ma argues that generative image editing is too broad to ship as a general promise. Teams need to identify specific user needs, guide people toward reliable workflows, and use multiple outputs where creative variation is acceptable. His practical advice is to build and speed up evals, then replace oversized models with distilled, efficient, or traditional solutions when possible.

## Key ideas
### Google Photos built computational photography around the compute already present on phones
[01:17](https://www.youtube.com/watch?v=C13jiFWNuo8&t=77s)
The editing team started in 2018 with the idea that phones had weak camera sensors but substantial computing power. A single image could be processed into an HDR-like result instead of requiring multiple exposures, a tripod, and Photoshop. Google could also work across Pixel hardware, Edge TPU acceleration, and internal computer vision researchers. Kelvin Ma describes a shared C++ library used by Android, iOS, and web clients, with on-device inference through TensorFlow Lite, now called LiteRT. This approach let the team ship features for old photos and devices without depending on a server.

### On-device models make editing consistent, but they create size, protection, and evaluation work
[04:24](https://www.youtube.com/watch?v=C13jiFWNuo8&t=264s)
For portrait segmentation, Google used a relatively focused U-Net convolutional neural network. The model reliably returned an output, but that output could still be wrong around fine hair strands. The team needed post-processing to improve the mask. The model itself was 10 megabytes, too large to bundle casually in the application, so Google had to download it later and manage model delivery and intellectual-property protection. Ma compares model benchmarks to unit tests. They need to reflect real usage, detect regressions, and be maintained over time.

### Magic Eraser became a coordinated pipeline of models and graphics code
[07:14](https://www.youtube.com/watch?v=C13jiFWNuo8&t=434s)
Magic Eraser, launched in 2021, combined distractor detection, segmentation, inpainting, and custom GL rendering. The rendering code showed the mask, animated it away, and brought in the filled area so the operation felt like one edit. This system was more complicated than the earlier portrait feature, and models grew to hundreds of megabytes. Failure cases also became more visible. Inpainting objects in the foreground was difficult for the models available at the time.

### Generative editing expanded the product while moving difficult engineering work to servers
[10:28](https://www.youtube.com/watch?v=C13jiFWNuo8&t=628s)
Magic Editor used larger state-of-the-art models because the best versions could not fit on mobile devices. Google therefore had to plan server capacity for GPUs and TPUs, account for network quality and data-center load, and handle round trips that could span long distances. Automated testing also became harder because the models were too large for the normal regression suite. The team had to consider captured responses or a test server, neither of which was ideal. Ma describes this as a major change from a local-first system where users supplied the compute.

### A broad promise such as generative image editing has to become a small set of user problems
[12:28](https://www.youtube.com/watch?v=C13jiFWNuo8&t=748s)
Ma says users do not ask for 'generative image editing' as an abstract capability. They have a concrete need, such as moving an object, changing a dull background, or removing a drink and its reflection. Product teams identify those demands, researchers train or adapt models for them, and engineers build the UX around workflows that work reliably. This narrowing matters because a demo that works 5 percent of the time is very different from one that works 50 or 80 percent of the time. A product cannot ship on the strength of a single successful example.

### Magic Editor uses constraints and multiple outputs to manage model uncertainty
[15:17](https://www.youtube.com/watch?v=C13jiFWNuo8&t=917s)
Ma agrees that prompts should not become the user's main interface. Google tries to extract intent and guide people toward a result rather than asking them to write a detailed paragraph on a phone. For defined tasks, the team uses targeted models and interaction patterns. For more open-ended creative cases, it can return multiple responses because there is no single correct image. He calls hallucination useful in that setting, while still saying trust and safety work must prevent the worst cases and set realistic expectations.

### AI engineering reduces the randomness that machine learning adds to software systems
[17:59](https://www.youtube.com/watch?v=C13jiFWNuo8&t=1079s)
Ma's working definition of AI engineering is software engineering with machine learning added. Models bring randomness and unpredictable edge cases, so engineers need to reduce that variability with evals, product constraints, and repeatable workflows. He recommends making evals fast enough to run often. Once a large model proves the product idea, teams should consider distillation, a more efficient model, or a traditional engineering solution. Faster iteration gives the team more chances to improve the product before launch deadlines.

### Google Photos is rebuilding its editor around AI and deterministic tools working together
[19:15](https://www.youtube.com/watch?v=C13jiFWNuo8&t=1155s)
At the Google Photos 10-year anniversary, the company announced a ground-up rebuild of the editor with an AI-first design. The editor is intended to surface relevant edits when a user taps an image, let the user adjust them, and use AI where helpful. Ma also describes the reverse relationship: AI can call deterministic tools to produce better edits. He is uncertain whether future progress will bring increasingly capable on-device models or keep the best capability in larger server models. His advice remains stable either way: iterate quickly and maintain strong benchmarks so the team knows what changed.

## Notable quotes
- "We want to use that to make really great image edits for any images." (01:40)
- "The benchmark is the equivalent of unit testing for your model and you need to maintain it and it takes time." (06:27)
- "You cannot ship a product that is 5% reliable if it's 50% reliable." (15:03)
- "The hallucination is a feature." (17:08)
- "To me AI engineering is just software engineering but with machine learning or ML on top." (17:59)

## Tools & references mentioned
- Google Photos
- Magic Editor
- Magic Eraser
- Pixel
- Edge TPU
- Android
- iOS
- TensorFlow Lite
- LiteRT
- U-Net
- OpenGL
- Halide
- DALL-E
- ChatGPT
- Gemini
- Gemma Nano
- DeepMind
- XKCD

## Who should watch
- You are shipping computer-vision or generative features in a consumer app and need to understand the cost of model size, device variation, and edge cases.
- Your prototype works in a demo, but you need to turn it into a narrow workflow with measurable reliability and a manageable testing setup.
- You are deciding whether a large model should remain in production or be replaced with a smaller model, a distilled model, or conventional code.

## Related talks

- [Prompt to Pipeline: Building with Google's Gen Media Stack](https://aietalks.com/talks/prompt-to-pipeline-building-with-googles-gen-media-stack) (Paige Bailey, Guillaume Vernade & Ian Valentine, Google DeepMind, 1:54:35)
- [Build & deploy AI-powered apps](https://aietalks.com/talks/build-deploy-ai-powered-apps) (Paige Bailey, Google DeepMind, 1:03:20)
- [Productionizing GenAI Models](https://aietalks.com/talks/productionizing-genai-models) (Lukas Biewald, Weights & Biases, 22:36)
- [The State of Generative Media](https://aietalks.com/talks/the-state-of-generative-media) (Gorkem Yurtseven, FAL, 17:14)
- [Lessons from building GenAI based applications](https://aietalks.com/talks/lessons-from-building-genai-based-applications) (Juan Peredo, 33:13)