# The AI developer experience doesn't have to suck: why and how we built Modal

Eric Bernhardson, Modal | AI Engineer Summit 2025 | 21:38

Source: https://www.youtube.com/watch?v=qeDPKbWjsuk
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/the-ai-developer-experience-doesnt-have-to-suck-why-and-how-we-built-modal
Published: 2025-02-22
Tags: deployment, developer-experience, inference, latency

## TL;DR
- Modal turns Python functions into cloud functions that can run on CPUs, GPUs, and large numbers of parallel containers.
- Eric Bernhardson argues that fast feedback loops require containers to start in the cloud almost as quickly as local code runs.
- Modal built its own scheduler, file system, image storage, and container-start optimizations because Kubernetes and Docker could not meet its target experience.

## Summary
Eric Bernhardson explains why Modal built a managed infrastructure platform around fast iteration on compute-heavy Python applications. Modal lets developers turn functions into serverless cloud functions, define their container environment in Python, select GPUs such as H100s, fan out work across thousands of containers, and deploy persistent endpoints. The central engineering problem is container cold start. Modal uses content-addressed storage to deduplicate image data, lazy loading to avoid fetching unused files, prefetching based on task traces, and snapshots of CPU memory. These techniques reduce the cost of starting Python environments with many sequential file accesses and allow large model environments to start within seconds. The platform also pools GPU capacity across users and regions, which supports usage-based billing without customer capacity planning. Bernhardson is clear that Modal is infrastructure for arbitrary code, rather than an AI model API, and that this flexibility leaves more responsibility with developers.

## Key ideas
### Fast feedback loops are the reason to rethink cloud infrastructure
[01:54](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=114s)
Bernhardson says developer productivity depends on a fast feedback loop. Cloud systems provide much more compute, but they can make iteration slower than local development. Modal started from the goal of taking code written on a laptop, running it in a remote container, and returning results quickly, including when the container needs a custom image or a GPU. He says starting containers in a distributed system became a deep engineering problem, leading the company to build its own scheduler, file system, and other foundational components.

### Modal exposes arbitrary Python code instead of a fixed AI model API
[00:41](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=41s)
Modal is aimed at high-code applications where developers write their own models or control how existing models fit into a workflow. Bernhardson compares it with Kubernetes or AWS Lambda because it can run arbitrary containers and code. The current interface is Python, chosen because Python dominates data, AI, and machine learning work. Examples include diffusion inference, medical-image processing, video computer vision, protein folding, language-model fine-tuning, embeddings, and batch jobs. Developers have more control, with more responsibility, than they would get from a narrow model API.

### The Python interface connects local iteration to remote compute
[04:43](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=283s)
A Modal decorator turns a Python function into a cloud function. Running the application from the command line sends the code to a container, executes it in the cloud, and streams output back. When the developer edits the function, Modal picks up the latest code and rebuilds the container automatically rather than requiring a slow manual image-build and log-download cycle. The same function can request an H100 by adding a GPU setting. Its image can be defined in Python, including a Debian Slim base and a pip installation of PyTorch.

### One function can scale from a test to thousands of parallel invocations
[08:32](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=512s)
Bernhardson demonstrates mapping a function over 5,000 inputs. Modal fans out the work by starting as many containers as possible, with the example reaching 18 containers during the live run. The same pattern can parallelize large batch jobs across GPUs or CPUs. Developers can also deploy a function to get a persistent endpoint, call it from another Python context, mount distributed file systems, create web endpoints, and schedule cron jobs. Idle containers shut down after the default period, while repeated calls can reuse a running container.

### Container images waste time through unused and duplicated files
[13:22](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=802s)
Bernhardson describes a container image as a root file system plus process isolation. Images often include data that an application never reads, such as Perl, man pages, local information, and time-zone data. Different images also contain many of the same files. Sending all of that data to a remote worker slows cold start. Modal therefore stores images as metadata pointing to content-addressed blobs, hashes the blobs for deduplication, and loads data lazily when it is accessed. He notes that AWS Lambda uses the same general technique.

### Python startup requires reducing thousands of sequential file accesses
[16:02](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=962s)
Python cold starts are especially sensitive to file-system latency. A container may need to read every Python module it imports, which can mean several thousand module accesses. Even a delay of several milliseconds per access can add seconds. Modal uses prefetching and task tracing, including historical information about which files a previous run accessed. Bernhardson says the company also built on container image builders and worked to bring startup below a second for suitable workloads.

### Memory snapshots can avoid rebuilding a large environment at startup
[17:03](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1023s)
Modal snapshots container images, but it also uses gVisor support for snapshots of CPU memory. Reverting a container to a saved RAM state can avoid loading the environment from scratch. Bernhardson says image caching and memory snapshots reinforce each other, and that Stable Diffusion can start in a couple of seconds even when its model weights are five or ten gigabytes. The company is also investigating GPU snapshotting, which he expects to make startup faster.

### Pooling GPU capacity makes usage-based serverless compute practical
[18:20](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1100s)
Fast container starts allow Modal to share expensive GPUs among users and give each application resources when it needs them. Bernhardson connects this to the original serverless promise: provision only what is used and pay for active capacity. Modal operates thousands of GPUs across cloud vendors and regions, scales pools up and down, and solves a mixed-integer programming problem to reduce total spend. It can keep a buffer of capacity so a request for 100 GPUs may be fulfilled quickly, without the customer buying or planning for that capacity.

## Notable quotes
- "In order to make Engineers fast and make them more productive you want to have this like super fast feedback loop that let you iterate on code very quickly." (01:54)
- "We couldn't use Kubernetes, we couldn't use Docker so we have to build a lot of this stuff ourselves." (12:48)
- "The promise of serverless was always don't provision more than you actually need." (18:41)
- "We're not an AI API, so to speak, you can run almost anything with Modal." (20:21)

## Tools & references mentioned
- Modal
- Kubernetes
- Docker
- AWS Lambda
- gVisor
- Linux
- PyTorch
- Stable Diffusion
- H100
- A100
- L4
- T4
- Spotify
- R2
- Eric Bernhardson

## Who should watch
- You are building AI or data applications and want to run your own Python code on cloud GPUs without operating a Kubernetes or Docker platform.
- Your team loses time to image builds, container startup, capacity planning, or slow feedback from remote jobs.
- You are designing serverless infrastructure and want concrete details about image deduplication, lazy loading, prefetching, and memory snapshots.

## Related talks

- [GPU Cloud Deployment Without Leaving Your IDE](https://aietalks.com/talks/gpu-cloud-deployment-without-leaving-your-ide) (Audrey Hsu, RunPod, 20:19)
- [Hypermode Launch](https://aietalks.com/talks/hypermode-launch) (Kevin Van Gundy, Hypermode, 05:03)
- [Keynotes & Multimodality Track](https://aietalks.com/talks/keynotes-multimodality-track) (Benjamin Duny & Simon Sturmer, AI Engineer & Chris Lattner, Modular & Auntie Aart & Mike Chambers, AWS & Alex Albert, Anthropic & Harrison Chase, LangChain & Rob Chandra, Substrate & Vic Kapati, M87 Labs & Ben Hilac, Dawn & Karan Goel, Cartesia & Chang She, LanceDB & Noah Schnapp, Character.AI & Steph Duenas & Quinn Dombrowski, Daily & Roman Huet, OpenAI & Brian Bischof, Charles Fry, Hamza Hussein, Jason Liu, Shreya Shankar & Eugene Yan & Thomas Dohmke, GitHub, 8:34:14)
- [Why Eval++ Is the Next Great Compute Primitive](https://aietalks.com/talks/why-eval-is-the-next-great-compute-primitive) (Sunil Pai & Matt Carey, Cloudflare, 24:51)
- [Multi model multimodal and multi agent innovations in Azure AI](https://aietalks.com/talks/multi-model-multimodal-and-multi-agent-innovations-in-azure-ai) (Cedric Vidal, Microsoft, 28:56)