LLMs improve Pinterest's ability to predict how relevant a pin is to a search query, especially as model size and capability increase.
2
Visual captions, board titles, search queries, and user engagement actions give Pinterest richer text annotations for each pin.
3
Pinterest distills an LLM teacher into a multilingual student model that can serve relevance scores efficiently at search time.
Summary
Pinterest Search handles more than six billion searches each month across over 45 languages and more than 100 countries. The speakers describe a relevance model that combines a query with pin content and predicts relevance on a five-point scale. A cross-encoder LLM performs well, but its cost makes direct online serving impractical. Pinterest instead fine-tunes a multilingual teacher on human labels, uses it to label examples from global daily search logs, and distills those labels into a two-tower student model. Pin embeddings are computed offline and query embeddings are computed online, with caching for repeated queries. The team also adds visual-language-model captions, board titles, high-engagement queries, and other user actions to represent pins. The resulting relevance-tuned embeddings are reused on related-pins and home-feed surfaces. The talk is practical about the tradeoff between a powerful relevance model and an architecture that can handle Pinterest's traffic.
Pinterest Search ranks pins through a five-level relevance prediction
Pinterest Search has query understanding, retrieval, ranking, and blending stages. The talk focuses on semantic relevance modeling in the reranking stage. Given a search query and a pin, the relevance model predicts a score on a five-point scale, from most relevant to most irrelevant. The initial LLM architecture is a cross-encoder. It combines the query and pin text before producing an embedding, which lets the model capture interactions between them. An MLP then maps that embedding to the five relevance levels. Pinterest fine-tunes open-source language models on internal data so they fit Pinterest's content and search behavior.
Larger LLMs improve relevance prediction in Pinterest's experiments
Pinterest compares LLM-based models with Search Sage, its in-house query and content embedding baseline. The LLM models substantially improve relevance prediction, and performance continues to improve as the team uses more advanced models and increases model size. The speakers give one comparison: an 8-billion-parameter Llama model improves performance by 12 percent over a multilingual BERT-based model and by 20 percent over the Search Sage embedding model. These experiments support the first lesson of the talk: LLMs are useful for predicting search relevance, provided the model is adapted to Pinterest's data.
Visual captions and user actions provide useful annotations for pins
To use language models for relevance prediction, Pinterest needs a text representation for every pin. The team combines the pin's title and description with a synthetic image caption generated by a visual language model. It also adds user-curated board titles and the search queries that produced the highest engagement for that pin. These user-action features give the model information about how people interpret and use the content. In ablation studies, the generated image caption is already a strong baseline. Adding more content features improves performance, and the final user-action features produce another measurable gain.
Pinterest uses a teacher model to create training data at global scale
The cross-encoder teacher is trained on a small set of human labels produced by annotators trained for specific segments. It is fine-tuned as a multilingual language model. Pinterest then samples daily search logs, which contain current queries, fresh pins, and activity from many countries and languages. The teacher assigns five-level soft relevance scores to these examples. This semi-supervised process expands the training data by about 100 times across domains, languages, and countries. It also addresses the limits of the human labels, which are concentrated in the United States. The student model learns from the teacher's scores and is the model served online.
A two-tower student model makes relevance scoring affordable
The production student model uses a bi-encoder rather than the teacher's cross-encoder. It embeds the pin and query separately, so it does not compute their full cross-interaction at serving time. Pinterest combines these language-model embeddings with Search Sage, Graph Sage, other query and pin embeddings, and text-match statistics such as BM25. Pin embeddings are inferred offline and cached across billions of pins. Pinterest recomputes them when their inputs change meaningfully. Query embeddings are generated online, and the short queries keep that work within a few milliseconds. Repeated queries are cached, producing about an 85 percent cache hit rate.
Teacher-generated labels transfer relevance gains beyond the United States
The production setup improves relevance measurements in the United States, Germany, and France segments discussed in the talk. Search fulfillment, which measures engagement with search actions, also increases outside the United States. The starting human-label data is mostly from the United States, while the daily logs contain international queries and content. The speakers attribute the transfer to the multilingual LLM teacher and its ability to produce useful labels across countries and domains. They present this as one reason to use semi-supervised labeling: Pinterest can extend a limited human-label set without manually annotating every market.
Relevance-tuned embeddings are reused across Pinterest surfaces
The student model produces pin and query embeddings as part of the relevance system. Pinterest also uses these representations for boards, based on their titles and related text. Because the embeddings are distilled from a large language model and remain part of a language-model-based student, the speakers find them useful as general semantic representations. They report improvements when adding them to related-pins and home-feed systems, as well as other Pinterest surfaces. This gives the relevance model a second use beyond the final reranking stage. The same representations can carry search-oriented meaning into other recommendation tasks.
Pinterest uses one multilingual model across languages
During the question period, the speakers explain that Pinterest uses the same model across languages rather than separate models for each language. The model is multilingual, so the team expects it to transfer information between languages. They also say visual captions are currently an effective way to represent image content as text. A fully multimodal embedding model is something Pinterest is exploring, while its current systems rely on in-house image-captioning models and text-based representations. The answer makes the current boundary clear: Pinterest gains multilingual coverage from one model and multimodal information mainly through generated captions.
"The LLM teacher model is trained on a small set of human label data that we get from human annotators who are trained in very specific segments."07:53
Who should watch
You are building search or recommendation ranking and need to balance cross-encoder quality with production serving cost.
Your relevance labels cover only a few markets, but your product has multilingual traffic and fresh content.
You want to reuse search-trained embeddings in related-content or feed-ranking systems.
