# AI Frontiers in Trust and Safety: Combatting Multifaceted Harm on Tinder at Scale

Vibhor Kumar, Tinder | AI Engineer World's Fair 2024 | 14:36

Source: https://www.youtube.com/watch?v=kwnCvA9l-TY
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/ai-frontiers-in-trust-and-safety-combatting-multifaceted-harm-on-tinder-at-scale
Published: 2024-12-02
Tags: cost, fine-tuning, guardrails, inference

## TL;DR
- Trust and safety at Tinder covers many harms with very different prevalence and severity, so detection needs separate models and policies.
- Human-AI dataset generation, parameter-efficient fine-tuning, and open-source serving tools can reduce model development from months to weeks.
- LoRAX makes it possible to serve many small LoRA adapters on shared base models, while heuristics and cascades control latency and cost.

## Summary
Vibhor Kumar explains how Tinder approaches detection across a long tail of trust and safety violations, from social handles in profiles to hate speech, harassment, scams, and underage users. Generative AI creates new problems by making spam, fake profiles, impersonation, and scam messages easier to produce. It also gives trust and safety teams useful pretrained models and open-source tools. Tinder uses GPT-4 with internal analytics data to mine candidate examples, then has people verify and correct the labels. The team fine-tunes open-source models with LoRA and QLoRA instead of calling an external API or training from scratch. In production, LoRAX serves many adapters on shared GPUs. Heuristics can reduce requests for common checks, and cascade classification can use a smaller model before calling a larger one. Kumar is direct about the operational challenge: harmful users adapt, so models need retraining and better generalization over time.

## Key ideas
### Trust and safety combines risk prevention with harm detection
[01:20](https://www.youtube.com/watch?v=kwnCvA9l-TY&t=80s)
Vibhor Kumar describes trust and safety using Del Harvey's breakdown: preventing risk, reducing risk, detecting harm, and mitigating harm. The purpose is to protect users and the companies that operate the products. Tinder focuses much of its work on detecting harm. The violations vary widely. Social media handles in profiles are common and relatively minor. Hate speech, harassment, and pig-butchering scams occur less often but can cause much greater harm. This range makes trust and safety an operational discipline that involves ethical judgment as well as model performance.

### Generative AI makes content pollution, impersonation, and scams easier to scale
[02:15](https://www.youtube.com/watch?v=kwnCvA9l-TY&t=135s)
Kumar says generative AI can rapidly create misinformation, propaganda, and low-quality spam, which can drown out genuine content. Deepfakes lower the barrier to impersonation and catfishing and can support interpersonal abuse such as revenge porn. Organized spam and scam operations can also generate profiles, images, and messages at high volume. Similarity matching and hashes become less useful when every profile and message is different. Kumar expects fraud teams to rely more on metadata linked to physical bottlenecks, including IP addresses, ISP information, and phone numbers.

### Pretrained open-source models shorten development and resist adversarial drift
[03:50](https://www.youtube.com/watch?v=kwnCvA9l-TY&t=230s)
Open-source models such as Llama and Mistral already provide semantic capability and broad language coverage. Kumar says a few-shot prompt can often detect violations reasonably well, while fine-tuning can reach state-of-the-art results and sometimes beat few-shot GPT-4 on downstream text detection tasks. Because teams fine-tune instead of training from scratch, development can move from months to weeks. This matters because trust and safety models degrade when bad actors change behavior after a rule is released. Kumar argues that the generalization of LLMs slows this degradation curve, especially when users introduce typos, mixed letters and numbers, or indirect language.

### A human-AI data pipeline turns internal platform data into training examples
[05:33](https://www.youtube.com/watch?v=kwnCvA9l-TY&t=333s)
Fine-tuning needs fewer examples than training from scratch, sometimes hundreds to thousands, but those examples must be high quality. Tinder can use GPT-4 to inspect internal analytics data and mine likely examples for a particular violation. Kumar prefers this hybrid approach to purely synthetic data because synthetic examples may not match the platform's real distribution and may ignore platform-specific context. He says the cost is inversely related to harm prevalence, and heuristics can restrict model calls to likely candidates. People then verify the mined examples, correct wrong labels, and make decisions on ambiguous cases.

### Fine-tuned local models provide control that external APIs cannot offer at Tinder's volume
[08:06](https://www.youtube.com/watch?v=kwnCvA9l-TY&t=486s)
Kumar gives two reasons Tinder does not rely directly on API models such as GPT-4 for production detection. The first is scale: real-time profile interactions create cost, latency, and throughput constraints. The second is maintainability. With its own fine-tuned models, Tinder controls the weights and can retrain when production performance falls without depending on changes to an underlying hosted model. Classification also exposes output probabilities, which lets the system assign confidence in a way familiar from traditional machine learning. Hugging Face libraries and configuration-driven tools further reduce the engineering needed for fine-tuning.

### LoRA and QLoRA make many specialized adapters practical
[09:37](https://www.youtube.com/watch?v=kwnCvA9l-TY&t=577s)
LoRA freezes the base model weights and learns only a small set of additional weights. Kumar says the adapter can be measured in megabytes, so fine-tuning can run quickly on one or a few GPUs and can make larger base models practical. QLoRA can allow fine-tuning on a single GPU, including very large models. Kumar's team can create separate adapters for categories such as hate speech, promotion, catfishing, and pig-butchering scams. The approach gives each harm type a specialized detector without requiring a separately trained full model for every category.

### LoRAX shares one base model across many real-time adapters
[10:14](https://www.youtube.com/watch?v=kwnCvA9l-TY&t=614s)
Tinder uses LoRAX, an open-source framework from Predibase, to serve many fine-tuned models on one GPU. Since each LoRA adapter is small, LoRAX can batch requests and shuffle adapters efficiently. Kumar says the marginal cost of adding an adapter to the same base model is virtually zero. Adding one in production involves storing its weights and changing the LoRAX client request. On A10 GPUs, Tinder can run seven-billion-parameter models at tens of queries per second with roughly 100 milliseconds of latency. He presents this as a way to cover many violation categories without horizontal scaling for every new detector.

### Heuristics and cascades keep expensive model calls within latency limits
[11:45](https://www.youtube.com/watch?v=kwnCvA9l-TY&t=705s)
Some high-frequency checks can be gated with simple heuristics. For social media handles in profiles, Tinder can run a model only when a bio contains a word outside a dictionary. The team is also exploring cascade classification. A smaller adapter first optimizes for recall, then a larger model is called only when the smaller model gives a high enough score. Classification and extraction are relatively cheap for LLMs because the output requires only one token or a few tokens instead of long autoregressive generation. Kumar reports near-100 percent recall for simpler social-handle detection tasks and substantial gains over older baselines on more semantic tasks.

## Notable quotes
- "Ultimately, T&S is about preventing risk, reducing risk, detecting harm, and mitigating harm." (01:20)
- "By fine-tuning these models, we can actually achieve state-of-the-art performance, in some cases better than few-shot GPT-4 performance on downstream textual detection tasks." (04:06)
- "The cost of doing this is inversely proportional to the true prevalence of the harm, but that cost is still pretty negligible." (07:27)
- "It means that we can train adapters for the many, many different types of trust and safety violations possible." (10:50)
- "Bad actors and other violative users always try to avoid detection." (13:06)

## Tools & references mentioned
- Del Harvey
- GPT-4
- Claude Opus
- Llama
- Llama 3
- Mistral
- Hugging Face
- Axolotl
- Ludwig
- Llama Factory
- H2O LLM Studio
- Predibase
- LoRAX
- LoRA
- QLoRA
- LaMDA
- OpenAI

## Who should watch
- You build trust and safety systems for a platform with many violation categories and need a way to cover the long tail.
- Your team is deciding whether to call hosted LLM APIs or fine-tune and serve open-source models in production.
- You need practical patterns for generating labeled data, adapting models cheaply, and routing real-time requests across detectors.

## Related talks

- [Critical AI Inference Your CIO Can Trust](https://aietalks.com/talks/critical-ai-inference-your-cio-can-trust) (Sahil Yadav & Hariharan Ganesan, Telemetrak, 19:04)
- [AI is the World's Largest Relationship Therapist](https://aietalks.com/talks/ai-is-the-worlds-largest-relationship-therapist) (Clay Cockrell & Tony Fabrikant, CoupleWork AI, 16:43)
- [How to Build Trustworthy AI](https://aietalks.com/talks/how-to-build-trustworthy-ai) (Allie Howe, Growth Cyber, 24:22)
- [Guardrails First: Engineering Member-Facing Health AI](https://aietalks.com/talks/guardrails-first-engineering-member-facing-health-ai) (Rashi Agrawal, Hinge Health, 21:49)
- [Building security around ML](https://aietalks.com/talks/building-security-around-ml) (Dr. Andrew Davis, HiddenLayer, 25:01)