Semantic IDs give recommendation systems content-aware item representations that help with new and infrequently interacted-with items.
2
LLMs can generate labels, query rewrites, and discovery content, but production systems work better when LLM output is combined with domain and behavioral data.
3
Unified recommendation models can reduce duplicated pipelines and improve several use cases, although tasks sometimes need to be split across multiple models.
Summary
This talk brings together examples from recommendation and search teams using language models. It begins with semantic IDs, which replace arbitrary item IDs with representations built from content and behavior. Examples from Kwaï and YouTube show how multimodal information can improve cold-start recommendations. The talk then covers synthetic data and labels. Indeed used a large model to label bad job recommendations before distilling it into a fast classifier, while Spotify used an LLM to generate exploratory search queries for podcasts and audiobooks. The final section examines unified models at Netflix and Etsy, where shared representations and rankers support several use cases. Pinterest and Instacart add practical lessons about using LLMs for relevance prediction, query understanding, rewrites, and discovery content. YouTube's LRM adapts Gemini to a video-specific language built from semantic IDs, but serving cost, freshness, and scale remain major constraints.
Semantic IDs reduce cold-start problems by encoding item content
Hash-based item IDs do not describe an item's content, so a new item has to be learned from scratch. This creates a cold-start problem and leaves long-tail items with too few interactions to learn useful representations. Kwaï's trainable multimodal semantic IDs combine visual, text, and audio embeddings, cluster those embeddings, and map the cluster IDs into trainable behavioral embeddings. The system used ResNet for visual content, BERT for video descriptions, and VGGish for audio. On a corpus of about 100 million short videos, the team learned 1,000 cluster IDs. The resulting IDs improved clicks and likes, increased cold-start coverage by 3.6%, and increased cold-start velocity.
LLM-generated labels become useful when distilled into fast production models
Indeed needed to filter poor job recommendations from email because unsubscribes were difficult to recover from and explicit negative feedback was sparse. Open models, including Mistral and Llama 2, produced generic answers. GPT-4 reached about 90% precision and recall, but took 22 seconds and cost too much. GPT-3.5 had poor precision, so the team fine-tuned GPT-3.5 and then distilled its labels into a lightweight classifier. That classifier reached 0.86 AUROC and was fast enough for real-time filtering. The system cut bad recommendations by about 20%, increased application rate by 4%, and reduced unsubscribe rate by 5%.
Spotify uses LLMs to extend proven query-generation methods
Spotify wanted users to discover podcasts and audiobooks after years of strong song and artist search. Its query recommendation system first generated candidates from catalog titles, playlist titles, search logs, and simple transformations such as adding 'cover' to an artist name. The LLM was used to add natural-language queries to this existing candidate set. Spotify ranked the new queries alongside immediate search results and displayed them in the search experience. This approach informed users about new product categories without requiring a banner. It increased exploratory queries by 9%. The speaker's advice was to keep conventional techniques that already work and use the LLM where it adds coverage.
Unified rankers share user and item learning across recommendation surfaces
Companies often maintain separate models for ads, search, home recommendations, item recommendations, add-to-cart suggestions, and thank-you pages. This creates duplicated engineering pipelines and makes improvements harder to transfer. Netflix's unified contextual ranker, called Unicorn, combines a user foundation model with context and relevance information. Its inputs can include user ID, item ID, search query, country, and task. Missing fields can be imputed from available context, such as using the current item's title when an item-to-item recommendation has no search query. The unified model matched or exceeded specialized-model metrics across several tasks. The value also comes from reducing technical debt and giving teams a shared foundation for later work.
Shared embeddings can combine relevance, personalization, and item quality
Etsy built unified query and product embeddings to handle both broad and specific searches in a changing marketplace. T5 models encode product descriptions, while query-product logs provide information about searches followed by clicks or purchases. The query and product encoders share representations for text tokens, product categories, and user location. User history adds personalization through searched queries, previous purchases, and other preference features. Etsy also added a quality vector containing signals such as ratings, freshness, and conversion rate to the product embedding. A matching query-side vector allowed the system to include quality in similarity scoring. The reported results were a 2.6% increase in conversion across the site and more than a 5% increase in search purchases.
Pinterest improves relevance with cross-encoder LLMs and multimodal annotations
Pinterest models search relevance as a five-level classification problem over a query and a pin. Its cross-encoder combines the query and pin text before an LLM produces an embedding for a multilayer perceptron. Fine-tuning open-source models on Pinterest data improved relevance prediction, with an 8-billion-parameter Llama 3 model performing 12% better than a multilingual BERT baseline and 20% better than Pinterest's search-stage embedding model. The team also used vision-language-generated captions, user-curated board information, and user actions as content annotations. Serving remained difficult because the model needed less than a second, and ideally about 400 to 500 milliseconds. The team used smaller models, gradual distillation, pruning, mixed precision, and attention sparsification.
Instacart combines LLM predictions with behavioral context for tail queries
Instacart's traditional query-to-category models worked on head and torso queries but had low coverage on the long tail. An LLM given only the query and taxonomy produced plausible categories that did not always match shopper behavior. For example, it interpreted 'protein' as chicken, tofu, or other protein foods, while Instacart users often meant protein shakes, bars, or supplements. The improved system supplied top converting categories and query-understanding annotations as context. For 'Werner soda', this helped identify ginger ale rather than only a broad fruit-flavored soda category. On tail queries, precision improved by 18 percentage points and recall by 70 percentage points. Instacart precomputed results for head and torso queries, cached them, and used existing or distilled models for the longer tail.
YouTube turns semantic video representations into a domain-specific language
YouTube's large recommender model adapts a Gemini checkpoint to recommendation tasks. Each video is represented using its title, description, transcript, audio, and video-frame information. The combined embedding is quantized with residual vector quantization into semantic IDs. These IDs become atomic tokens for a new language of YouTube videos. Continued pre-training links the IDs with English descriptions and teaches the model to predict masked videos from user watch sequences. The model can then generate video IDs for retrieval and reason over semantic relationships in a user's history. It produced distinctive recommendations for difficult cases, including related women's races after a user watched an Olympics highlights video. The main limitation was serving cost, which YouTube reduced by more than 95% before production use.
"The biggest limitation was that the serving costs are too high, especially for the scale that YouTube operates at with billions of users."1:18:21
Who should watch
You are building a recommender or search system with sparse feedback on new items and need content-aware representations.
Your team wants to use LLMs for labels, rewrites, or discovery features without putting a slow model directly in the online path.
You maintain many specialized ranking models and are deciding whether shared representations or a unified model could reduce duplicated work.
