Sampled profiling gives teams a low-overhead way to observe production workloads continuously.
2
GPU metrics can be correlated with CPU stack profiles to explain why a GPU is underused.
3
GPU time profiling measures how long individual CUDA kernels spend running on the GPU.
Summary
Matthias Loibl explains how continuous profiling can help teams understand GPU performance in production. Sampled profiling records stack information at a chosen rate, which keeps overhead low while revealing the recurring work that matters. Polar Signals uses Linux eBPF, so applications do not need code changes or manual instrumentation. The GPU preview collects utilization, memory, clock speed, power, temperature, and PCIe throughput through NVIDIA NVML. These metrics can be compared with CPU profiles to find periods when the CPU is busy loading data or calling CUDA while the GPU is not fully utilized. Loibl then introduces GPU time profiling, which measures the duration of CUDA kernels on the GPU and displays that time in a combined CPU and GPU view. The tooling runs on Linux, Docker, or Kubernetes, and works across languages and application types.
Loibl contrasts tracing, which records every event and creates substantial data and cost, with sampled profiling. The profiler samples for a chosen duration, such as 10 seconds, at a rate such as 20 or 100 times per second. At 100 samples per second, he says CPU overhead is less than one percent and memory profiling adds about four megabytes. Sampling can miss one-off events, but continuous collection eventually exposes recurring, relevant stacks. The goal is to understand the larger performance picture rather than every isolated execution.
Production profiling needs low overhead and no application changes
Loibl argues that a developer's machine does not show what happens in production, so profiling needs to run on real workloads. Polar Signals uses Linux eBPF, with the kernel doing the observation work. This means teams do not have to modify or instrument each application. Starting the profiler allows it to collect profiles from the applications running on the machine. That makes always-on profiling practical across a production environment while keeping the collection cost low.
GPU metrics reveal utilization and hardware bottlenecks
Polar Signals' GPU preview talks to NVIDIA NVML to collect measurements for both individual processes and the whole node. The view includes GPU utilization, memory utilization, and clock speed. It also includes power usage, the GPU power limit, temperature, and PCIe throughput. Loibl points out that sustained high temperature can cause the GPU to throttle. PCIe throughput can show whether the system is limited by data moving between the CPU and GPU. These measurements indicate where a period of low GPU utilization deserves investigation.
The GPU measurements can be correlated with the CPU stack profiles Polar Signals has collected with eBPF. A user can select a time range and view a flame chart for the period when GPU utilization drops. The chart may show Python calling CUDA, or a compiled language such as Rust integrating with CUDA. In many cases, the CPU is busy loading data instead of keeping the GPU supplied with work. The same profiling approach can show activity from Python, Ruby, the JVM, and other application environments.
Loibl introduces GPU time profiling as a new feature in the talk. When a CUDA stack is placed on the CPU, the system records the start time. It then records when the kernel terminates, which gives the duration that function spent on the GPU. This differs from only seeing CPU-side calls into CUDA. The resulting profile attributes GPU time to individual functions, allowing users to find which parts of the call stack consume time on the accelerator.
In the example, the profile starts with a Python main function and continues through libcudа. The width of each displayed stack reflects the time the associated function takes on the GPU. The leaf of each stack is the function that actually consumed GPU time, while the upper part shows the CPU call path that led to it. Colors identify different binaries running on the machine, such as Python and CUDA. The result connects application code with the GPU kernels it launches.
The profiler runs as a Linux service or Kubernetes DaemonSet
Teams can get started with a binary on Linux, run it with Docker, or deploy a Kubernetes DaemonSet. The Kubernetes setup uses a manifest and a token. Loibl says existing CPU and memory profiling customers are beginning to add GPU profiling to their platforms. He names TurboPuffer as an example of a company interested in improving the performance of its vector engine.
"The width of these stacks that we are seeing is the actual time that we had these functions take up in the GPU."09:42
Who should watch
You run GPU workloads in production and need to explain periods of low utilization.
Your team wants CPU and GPU performance data without adding instrumentation to each application.
You are evaluating continuous GPU profiling or need to identify CUDA kernels that consume the most GPU time.
