# 360Brew: LLM-based Personalized Ranking and Recommendation

Hamed Firooz & Maziar Sanjabi, LinkedIn AI | AI Engineer World's Fair 2025 | 22:00

Source: https://www.youtube.com/watch?v=U0S6CfzAY5c
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/360brew-llm-based-personalized-ranking-and-recommendation
Published: 2025-07-16
Tags: distillation, inference, latency, rag, reranking

## TL;DR
- LinkedIn built 360Brew as one foundation model for ranking and personalization across multiple surfaces and tasks.
- The team trains a 150-billion-parameter model, then distills it through smaller models because the large model cannot meet online recommendation latency and throughput requirements.
- Gradual distillation, pruning, mixed-precision quantization, and sparse attention reduced latency by 7x while increasing per-GPU throughput by 30x.

## Summary
Hamed Firooz and Maziar Sanjabi describe LinkedIn's effort to build one foundation model for ranking and recommendation instead of maintaining separate models for each task. Their approach converts member profiles, histories, and interactions into prompts, then trains a model through continued pretraining, fine-tuning, instruction fine-tuning, and alignment. The largest version, Blue XL, has 150 billion parameters. It improves cold-start behavior and can perform competitively on tasks outside its training domains, but it is too expensive to serve directly. The team therefore distills it through intermediate models, applies gradual pruning, uses mixed precision, and sparsifies attention. These steps produced a 7x latency reduction and a 30x increase in throughput per GPU across several releases. The speakers are direct about the engineering cost: automation is needed to run experiments, and context selection must balance user freshness, positive and negative examples, and KV-cache constraints.

## Key ideas
### A single foundation model can replace many separately trained ranking systems
[01:00](https://www.youtube.com/watch?v=U0S6CfzAY5c&t=60s)
Firooz says recommendation systems are embedded in feeds, job search, shopping, and other everyday products, but they are usually trained for individual tasks. That makes them disjoint, limits the architectures teams can use, and forces organizations to roll out models one at a time. LinkedIn's mission with 360Brew is to build one large language model with a holistic understanding of a member's journey and use it across personalization tasks. The intended model also supports new surfaces without starting a full data-collection and deployment process for each one. The same model can be prompted with a user, an entity, and a task, then asked to judge relevance.

### Zero-shot prompting, in-context examples, and instructions target new and cold-start cases
[01:51](https://www.youtube.com/watch?v=U0S6CfzAY5c&t=111s)
The team wanted 360Brew to work on a new problem or surface without building a new ranking model from scratch. Firooz describes a zero-shot interface where a developer specifies the task, the entity, and the user, then asks for their relevance. In-context learning is aimed at cold-start users. A small number of examples, or a description of what the user may like, can give the model enough information to make recommendations. The system is also intended to accept direct member instructions, such as niche interests or topics the member wants to explore, and use those instructions to find relevant information.

### Promptification turns LinkedIn's user data into a language-model input
[03:41](https://www.youtube.com/watch?v=U0S6CfzAY5c&t=221s)
Sanjabi calls the data preparation step the "magic of promptification." The team takes member history, profile information, and past interactions, then converts them into a prompt. The prompt contains an instruction, member-profile details, and examples of information previously shown to the member. It ends with a question about what the user will do with a new item or piece of information. This format lets the model generalize across instructions rather than treating every recommendation task as a separate model. An off-the-shelf LLM can do some of the work, but the speakers say it needs training to perform well.

### The largest model is a teacher for smaller production models
[04:31](https://www.youtube.com/watch?v=U0S6CfzAY5c&t=271s)
The model-development pipeline begins with an open-source model, followed by upcycling to control model size, throughput, and quality. LinkedIn applies continuous pretraining, fine-tuning, instruction fine-tuning, and alignment before producing Blue XL, a 150-billion-parameter model. That model delivers the best quality in their setup, but recommendation systems need high throughput and low latency, so it cannot directly serve online traffic. The team distills it into smaller models, down to a 3-billion-parameter model that can be productionized. Their experiments show that training a small model from scratch is weaker. The recipe is to go big first and then go small.

### Data, model size, and context length each improve quality up to a limit
[06:32](https://www.youtube.com/watch?v=U0S6CfzAY5c&t=392s)
Sanjabi describes three ways to improve the model. More logged user behavior improves performance, with useful history extending across periods such as six months or a year. Increasing the model size also improves quality in their mixture-of-experts experiment, including a comparison from 7B to 8x22B. Context length matters because it controls how much of a user's history reaches the model. Performance rises as more history is provided, then falls at longer lengths. The team attributes that decline to poor generalization of the model used in the experiment at long context lengths, rather than to the extra context becoming inherently less informative.

### 360Brew helps most when users have little interaction history
[08:47](https://www.youtube.com/watch?v=U0S6CfzAY5c&t=527s)
The cold-start comparison measures the gap between 360Brew and production models for users with different amounts of interaction history. The gap grows as the number of interactions decreases, including the range below five interactions and the range below 100 interactions. Firooz attributes the improvement to the world knowledge the model brings into the recommendation system. The model also generalizes to four tasks that were completely outside its training domains. On those tasks, it can match or beat production models trained specifically for each task. The speakers connect this result to faster rollout of new LinkedIn features and surfaces.

### Serving requires gradual compression rather than one aggressive reduction
[10:53](https://www.youtube.com/watch?v=U0S6CfzAY5c&t=653s)
LinkedIn recommendation systems can have tens of thousands of queries per second and require latency around 400 to 500 milliseconds or less. Firooz says the team uses several efficiency levers, while keeping the large teacher model in the training path. Distillation proceeds step by step, for example from 150B to 8B, then 3B, then 1B, with each smaller model distilled from the previous one. Pruning follows the same gradual pattern. The team removes transformer heads or MLP capacity in stages, distills again, and repeats. Aggressive pruning at the beginning can reduce model quality by up to 1 percent, while gradual pruning can avoid measurable information loss in their experiment.

### Mixed precision and sparse attention reduce serving cost
[12:51](https://www.youtube.com/watch?v=U0S6CfzAY5c&t=771s)
The serving stack uses FP8 for activations and model parameters, but the language-model head remains in FP32. Firooz says using lower precision throughout causes the output numbers to collapse and harms calibration, which makes it harder to distinguish recommended items. The team also sparsifies attention because every item does not need to attend to every other item. When scoring many candidate items, the candidates can attend to historical user and profile information without attending to one another. This lets the system score up to 500 items in a query segment while avoiding unnecessary attention work. Across four or five releases, the combined changes cut latency by 7x and raised per-GPU throughput by 30x.

### History selection must balance freshness, relevance, class balance, and caching
[19:14](https://www.youtube.com/watch?v=U0S6CfzAY5c&t=1154s)
For long user histories, LinkedIn tested retrieval of the most relevant historical items and chronological ordering with time decay. Chronological order was good enough for most applications because recommendation systems favor fresh activity. The team also has to decide how many positive and negative examples to place in the prompt and how much information to include. Serving adds another constraint: a smarter history arrangement can interfere with KV caching. Sanjabi says making the context selection smarter than chronological order is therefore an engineering and serving-design problem, rather than an obvious improvement.

## Notable quotes
- "You have to first go big and then go small." (06:18)
- "The recipe here is also do the gradual pruning." (12:46)
- "If you put everything together, we can have a significant reduction in the latency." (15:00)
- "I think automation is the key if you want to really optimize for these models." (20:57)

## Tools & references mentioned
- LinkedIn
- LinkedIn AI
- Meta AI
- Hateful Memes benchmark dataset
- Blue XL
- NL Web
- Microsoft
- FP8
- FP32
- SGLang
- vLLM

## Who should watch
- You are building ranking or recommendation models for several products and want to understand the case for a shared foundation model.
- Your team has a large language model with good offline quality but needs to reduce its serving latency and GPU cost.
- You need to handle cold-start users, new recommendation surfaces, long user histories, or experiments across distillation and quantization.

## Related talks

- [Improving Recommendation Systems and Search in the Age of LLMs](https://aietalks.com/talks/improving-recommendation-systems-and-search-in-the-age-of-llms) (Eugene Yan, Amazon, 20:54)
- [Personalization in the Era of LLMs](https://aietalks.com/talks/personalization-in-the-era-of-llms) (Shivam Verma, Spotify, 20:12)
- [LLM Recommendation Systems (RecSys)](https://aietalks.com/talks/llm-recommendation-systems-recsys) (, 4:06:11)
- [How fast are LLM inference engines anyway?](https://aietalks.com/talks/how-fast-are-llm-inference-engines-anyway) (Charles Frye, Modal, 16:07)
- [Transforming search and discovery using LLMs](https://aietalks.com/talks/transforming-search-and-discovery-using-llms) (Tejaswi Tenneti & Vinesh Gudla, Instacart, 21:10)