Snorkel AI develops domain-specific data for fine-tuning models because general-purpose models often miss enterprise requirements for accuracy, reliability, latency, and cost.
2
Its distributed training stack uses PyTorch, Horovod, Azure virtual machines, and shared network storage to run jobs from one node to dozens of nodes.
3
GPU utilization, network placement, storage throughput, and flexible access to compute determine whether large training jobs run efficiently.
Summary
Humza Iqbal explains how Snorkel AI develops data for enterprise models and runs its research workloads on Azure AI infrastructure. General-purpose models can be useful, but customers such as banks and insurers need better accuracy, reliability, latency, and cost. Snorkel therefore builds domain-specific data for fine-tuning, evaluation, alignment, and multimodal model development. For distributed training, the team uses PyTorch with Horovod across Azure virtual machines, while a shared network file system provides common access to training data and checkpoints. Iqbal describes practical failure points, including poorly chosen batch sizes, network distance between data and compute, slow storage reads, and low GPU utilization. He also compares Azure A100 and H100 deployments, saying that two H100s delivered better training and inference than four A100s at roughly the same cost in their test. The resulting faster iteration lets the team generate more synthetic data and evaluate models more often.
Enterprise models need domain-specific data to reach production requirements
Humza Iqbal says Snorkel AI helps enterprises develop data for fine-tuning models because out-of-the-box models rarely meet the accuracy, reliability, latency, and cost requirements of customers such as banks and insurance companies. Claude, GPT-4, and Gemini can cover part of the problem, but they do not provide the final level of performance needed for deployment. Retrieval-augmented generation is useful for connecting models to enterprise knowledge, yet Iqbal describes it as a starting point. Large instruction-tuning datasets may still miss the narrow slice of information, such as a particular bank policy, that a model needs.
Snorkel combines subject-matter experts with programmatic data development
Snorkel's research focuses on helping enterprises develop data for custom AI models. Iqbal describes three directions: keeping subject-matter experts involved while getting more value from their time, making data development programmatic, scalable, and auditable, and evaluating models with continuously updated, domain-specific benchmarks. Experts know the details needed for useful feedback, but manual work alone does not scale. General benchmarks also miss the concerns of individual industries. A bank may need to measure policy and compliance behavior rather than performance on a broad academic task.
Evaluation must test the domain and behavior that matter to the customer
Iqbal describes active work on fine-grained evaluation, enterprise alignment, and multimodal alignment. Long-context tests such as the needle-in-a-haystack setup can show whether a model finds inserted information, but he says they do not necessarily capture long-context behavior in other domains. Snorkel is also studying whether models follow company goals and regulations. In multimodal work, the team is using large vision-language models to generate synthetic data without manual annotation, then using that data to train downstream models. The goal is a loop from domain-specific data generation to model training.
The distributed training stack connects Horovod, Azure VMs, and shared storage
Snorkel uses PyTorch as its machine learning framework and Horovod for communication between nodes. The workloads run across Azure virtual machines with NVIDIA A100 or H100 GPUs. Horovod sends gradients between the distributed copies of the model. The machines read and write through a shared network file system, which gives each VM access to training data and model checkpoints as though the files were local. Iqbal says the team has used this setup for jobs ranging from one node to dozens of nodes, including DPO alignment, preference optimization, multimodal training, and multimodal inference.
Batch size and data placement can decide whether more GPUs help
Iqbal says the team had to match the number of nodes to its intended batch size. In computer vision training, too few nodes could prevent the desired batch size, while too many could create an overly large batch or leave nodes underused. Network placement is another concern. Data and compute should be close together because gradient communication and data transfer can become bottlenecks when they cross distant regions. Storage read speed matters during training too. If the network file system cannot supply data quickly enough, the model waits instead of computing.
Iqbal recommends checking GPU utilization with nvidia-smi when a distributed job is slow. Low utilization can point to different causes depending on the setup. In a multi-node job, the team can investigate network communication. On a single node, the problem is more likely to be data loading. He also stresses reliability, flexibility, and manageability. Snorkel needed to scale compute up or down over a long period, rather than accept a large allocation that expired after a fixed month or two. That access pattern fit research experiments better than a short, fixed training window.
H100s improved performance per dollar in Snorkel's comparison
Snorkel compared two H100s with four A100s at roughly the same cost. Iqbal says the H100 setup performed better for both training and inference, which made it better per dollar in that comparison. He valued the simple hardware change because it delivered an immediate improvement without requiring every inference optimization first. Faster inference allowed the team to process more synthetic data and improve model accuracy through a faster iteration cycle. He also notes that the comparison kept the relevant batch sizes fixed between the hardware configurations.
Snorkel plans to expand synthetic data, taxonomies, and multimodal retrieval
Iqbal outlines several areas Snorkel is actively pursuing. The team wants to use preference signals more programmatically for data synthesis and curation. It also wants scalable entry points for data development based on rationales and custom taxonomies. A third area is improved multimodal retrieval, evaluated on domain-specific datasets so that retrieval models can be scaled with evidence from the target domain. These projects continue the same emphasis on developing and measuring data for specific enterprise use cases.
"It just enables us a flywheel of faster iteration which is super critical for being able to do more development."16:34
Who should watch
You are choosing GPU instances and storage for distributed model training and need practical checks for batch size, network placement, and data loading.
Your research team needs compute that can expand and contract over a long period instead of a short fixed allocation.
You are building enterprise or multimodal models and need domain-specific data, evaluation, and synthetic-data workflows.
