Krea 2 was trained from scratch on thousands of GPUs, with separate raw and Turbo checkpoints released for use.
2
Tensor core, temperature, InfiniBand, and NVLink metrics gave Krea visibility into failures that ordinary GPU dashboards missed.
3
Krea's Kubernetes setup gives training priority while moving inference to external providers and migrating it back gradually when GPUs become available.
Summary
Gabriel Jorge Menezes describes the infrastructure behind Krea 2, a diffusion transformer trained from scratch on thousands of GPUs. At larger cluster sizes, runs crashed more often, including silent failures where dashboards stayed green. Krea responded with strict operational rules: remove GPUs above 78 degrees, track tensor core use instead of trusting GPU utilization, and collect InfiniBand and NVLink data that standard NVIDIA tooling does not expose. Frequent checkpointing kept failed runs recoverable, supported by a filesystem that could write a terabyte in less than 30 seconds. Training and production share one Kubernetes cluster, with gang scheduling and training given higher priority. A virtual Kubelet node sends evicted inference workloads to outside GPU providers. Taints and a descheduler then move workloads back gradually, avoiding a production outage. The talk is practical and candid about accepting crashes, replacing unreliable hardware, and letting Kubernetes recreate failed pods.
Krea 2 was trained from scratch for creative exploration
Gabriel Jorge Menezes introduces Krea 2 as a model trained entirely in house, without a base checkpoint. Krea built it to help creatives explore unusual image compositions and styles rather than produce only conventional AI images. The project released two checkpoints: a raw pretrained checkpoint for people who want to post-train it, and a much faster post-trained Turbo version. Menezes says the model is open source through Hugging Face and GitHub, and is also available in production on Krea.ai. The infrastructure challenge came from training this model on thousands of GPUs connected with InfiniBand.
Larger clusters made failures more frequent and often silent
Small experiments could run for days, but failures became more common as the team scaled through 128, 256, 512 GPUs and beyond. Some failures involved InfiniBand timeouts, while dashboards continued to show healthy metrics. Krea initially swapped nodes and changed hardware whenever a run failed. The team later found that restarting on the same machines, with the same code and data, could produce a run lasting 12, 16, or 24 hours. Menezes points to a Meta paper that gives a rough estimate of expected failures, although Krea's runs lasted much less than the paper's examples. Runs of less than eight hours made pretraining difficult because the GPUs stopped making progress.
Temperature rules were stricter than debugging individual GPUs
Menezes says metrics gave the infrastructure team the visibility needed to support researchers. GPU temperature was one simple but effective measure. A single GPU that runs warmer than its peers can throttle, slow training, and create unstable behavior across the run. Krea's rule was to remove any GPU above 78 degrees Celsius instead of trying to diagnose or repair it in place. The provider was asked to replace the card. Menezes presents this as a time-saving operational decision: a warm card can affect the whole training job, so the team removes it before it causes harder-to-explain problems.
Tensor core utilization exposed work that GPU utilization hid
The standard GPU utilization metric stayed at 100 percent during pretraining, but Menezes says that number did not mean the GPUs were being used efficiently. It measured how long the GPU was doing work, without showing whether that work was effective. Krea used tensor core utilization as a proxy for how much of the tensor hardware was actually being used. The team watched this number rise as image resolution increased through 128, 256, 512, and 1024 pixels during pretraining, mid-training, and post-training. The resolution changes therefore gave them a more useful view of how much computation the workload was driving.
InfiniBand and NVLink monitoring had to be built in house
Standard NVIDIA DCGM metrics did not export the InfiniBand data Krea needed, although they exposed some NVLink information. Menezes says cross-node communication caused most of the team's failures, so InfiniBand monitoring became one of the most important parts of the dashboard. Krea collected message wait time on the fabric, error counts, packet counts, and other exported values with custom tooling. The team also collected NVLink errors to find problems inside a single machine, where GPUs could appear healthy while the node was still failing. Once a node showed these errors, Krea replaced the machine. Menezes says the custom collection was straightforward to build.
Frequent checkpointing made short training runs recoverable
Because training crashed often, Krea used checkpointing aggressively. Menezes says the team's initial Ceph setup was unreliable and caused them to lose trust in the data. He recommends a paid filesystem when the budget allows, because the team could measure its performance and rely on it. Krea's worker cluster reached about 1.8 terabytes per second for reads and nearly one terabyte per second for writes. The filesystem could write a terabyte in less than 30 seconds, so the team checkpointed every 20 or 30 minutes without stalling training. This let them recover progress instead of losing the work from a failed run.
Gang scheduling gives training priority over production inference
Krea wanted researchers to submit training jobs without choosing GPUs themselves. Jobs enter a queue, and gang scheduling makes the required group of resources available together. The system has workload priorities, followed by normal Kubernetes priorities. Training pods receive higher priority, so they can evict inference pods from the shared cluster. Menezes says this reflects Krea's resource choice: production must remain available, but the GPUs produce more value for the company while training. The queue's resource definitions are manually maintained, which becomes inconvenient as nodes enter maintenance or leave the cluster. Menezes also mentions Kubernetes 1.15 as having similar gang scheduling support built in.
A virtual Kubelet node moves evicted inference elsewhere
Krea keeps production available by flipping inference traffic to another cluster or to rented GPUs when training needs the main cluster. The mechanism uses the open source Virtual Kubelet project to create a fake Kubernetes node. Kubernetes schedules a pod onto that node, and Krea's provider layer translates the pod into the API of an external GPU provider. A reconciliation process keeps the Kubernetes and provider states aligned. If something fails, Krea marks the pod as failed and lets Kubernetes create a replacement. Menezes says this avoids writing custom recovery logic for every failure on either side.
Taints and gradual migration prevent wasted GPUs and outages
Krea uses Kubernetes taints and pod tolerations to control when inference can run on the main cluster. When GPUs are available, the system taints the node so workloads that should use the GPUs do not consume them. When training takes all the GPUs, the taint is removed and a new pod can schedule there. After training ends, the system adds the taint again and a descheduler moves inference workloads back. Krea avoids a no-execute taint because it would evict everything at once and could take production down. The gradual migration keeps the site running while reducing the time that rented GPUs are used. Menezes also says diffusion transformer inference can run on almost any GPU, even unreliable hardware.
"Metrics are everything. That's how I can support my researchers. That's how I have visibility in the system."04:18
Who should watch
You run multi-GPU or multi-node training jobs and need practical rules for diagnosing communication failures.
Your training and inference workloads compete for the same Kubernetes cluster, and you need a way to evict and restore production safely.
You are building infrastructure for diffusion transformers and want guidance on checkpoint frequency, hardware replacement, and GPU monitoring.
