RunPod Flash deploys an async Python function to GPU cloud with an `@flash.endpoint` decorator, while the surrounding code stays local.
2
Hot reload lets developers change models and parameters from their IDE without rebuilding and uploading a Docker image.
3
RunPod recommends pods for early experiments and serverless when workloads need large-scale autoscaling across data centers.
Summary
Audrey Hsu explains how RunPod's Flash Python SDK shortens the development loop for GPU inference. The usual process requires committing code, building a Docker image, pushing it to a registry, loading it on a server, allocating a GPU, and then testing. Flash lets a developer add `@flash.endpoint` to an async Python function and run the GPU portion in the cloud from a local development server. File changes are repackaged and pushed automatically. Hsu demonstrates this with Stable Diffusion XL Turbo, then swaps in DreamShaper without leaving the IDE. A second demo chains Qwen 3, DreamShaper, and Nano Banana 2 into a prompt, image generation, and photo-composition pipeline. She also explains the difference between pods and serverless, including request-based billing and worker autoscaling.
RunPod removes infrastructure setup from GPU application development
Audrey Hsu describes RunPod as an AI cloud infrastructure company that provides GPUs and compute while letting developers bring their own code and models. She says teams often spend more time dealing with CUDA versions, PyTorch compatibility, GPU SKUs, and related bugs than working on their models or applications. RunPod offers several deployment modes. Pods provide persistent or on-demand VM environments with reserved GPUs and per-second billing. Serverless scales workers up and down with request load. Clusters target multi-node training, while Hub provides pre-vetted repositories for projects such as ComfyUI, Stable Diffusion, and vLLM.
The normal inference iteration loop is slow because every test passes through deployment steps
Hsu describes the process developers typically repeat while testing an inference function. They commit the code, push it to GitHub, build a Docker image, pull it from a container registry, load it onto a server, allocate a GPU, and only then find out whether the change works. Each new experiment repeats the same sequence. This creates a delay between changing model code and seeing its output. Flash is designed to remove that cycle by allowing the function to run on GPU cloud directly from the developer's local environment.
A decorator sends only the GPU function to the cloud
Flash uses a regular async Python function with an `@flash.endpoint` decorator. Hsu says the decorator packages everything inside the decorated function and deploys it to a GPU cloud. The main function and helper functions around it continue to run locally. This split lets a local development server receive requests while the expensive GPU work runs remotely. Flash also watches for file changes. When application code changes, it repackages and pushes the updated function so the developer can test again without manually rebuilding a container.
The first demo uses a local server to call Stable Diffusion on a remote GPU
Hsu demonstrates an image-generation function that loads PyTorch and the pre-trained Stable Diffusion XL Turbo model. The function saves the generated image and returns it as base64. She starts Flash's local development server, sends it a POST request, and passes a prompt about cats flying through a cloudy London sky. The endpoint queues a job and starts a worker. The first result produces abstract-looking cats, which gives Hsu a reason to change the model during the live demonstration.
Model swaps happen from the IDE without a container rebuild
After rejecting the first output, Hsu comments out the Stable Diffusion XL Turbo code and switches the function to DreamShaper, a fine-tuned model based on Stable Diffusion 1.5. She changes the inference settings, including the number of steps, then sends the same request again. Flash detects the code change and updates the remote function. Hsu contrasts this with committing, rebuilding a Docker image, uploading it, and allocating GPU infrastructure. The model change happens from the same IDE where she wrote the function.
A useful GPU application includes orchestration across several model endpoints
Hsu says the value of a developer tool becomes clearer when an application coordinates several model calls. Her prepared pipeline asks Qwen 3, hosted on a public endpoint, to generate image prompts. Those prompts go to DreamShaper running on a RunPod endpoint. A further step sends the generated images to Nano Banana 2, a Google model that composes photos together. In the demo, the pipeline creates prompts for two men with glasses walking in cloudy London weather, generates three images, and combines them into a final photo.
Serverless billing follows worker uptime during requests
Hsu opens the RunPod console to show the endpoint and its workers after requesting three images. Several workers are provisioned, and three are running for the three photo requests. She explains that the charge is based on how long each request is running, which she describes as the worker's uptime during the request. She gives an H100 price of $0.00116 per second. Serverless has a premium compared with pods because it provides scaling, while pods do not provide that same scaling behavior.
Pods fit experiments while serverless fits large variable workloads
RunPod's recommendation depends on the workload stage. Developers who are experimenting may need only one or two GPUs at a time, so Hsu suggests starting with pods or using a very low serverless worker count. Serverless becomes more appropriate when an application needs hundreds of workers on hundreds of GPUs. It can distribute those workers across data centers for better availability and scale them down when requests stop, so idle workers are not charged in the same way.
"The problem that Flash is trying to solve here, and Flash is our Python SDK, is that we want to eliminate all of that iteration cycle so that you can deploy your function on a GPU right from your local development environment."06:20
Who should watch
You are building GPU inference features and spend too much time rebuilding containers between model changes.
Your application calls several hosted or self-hosted models and you want to test the orchestration code locally.
You are deciding whether a persistent pod or autoscaling serverless workers fit your workload.
