Open models are trained through pre-training, instruction tuning, and preference alignment, with each stage using different data and compute.
2
Structured generation can force open models to produce valid JSON, regular-expression matches, and other formats while reducing wasted tokens.
3
Domain-specific training, long context, careful fine-tuning, and model merging can adapt open models to finance and other specialized tasks.
Summary
This Open Models track covers the work needed to make open language models useful in real applications. The first talk explains the three-stage training process and the trade-off between model performance and running cost, then introduces a code model designed for completion and chat. Remy presents Outlines, a Python library that constrains token generation so models return valid structures instead of verbose or malformed output. Sandra describes Cohere's Command R models, their retrieval and tool-use behavior, fine-grained citations, and multi-step API. Leo explains Gradient's approach to finance models, using domain data and extended context to reduce hallucinations. Daniel focuses on bugs that can break Llama 3 fine-tuning, including double BOS tokens, incompatible chat templates, untrained tokens, and identical padding and end-of-sequence tokens. Maxim covers supervised fine-tuning, preference alignment, data preparation, quantization, and model merging techniques such as SLERP, DARE, passthrough, and mixture-of-experts merging.
LLM training has three stages with sharply different costs
The first speaker describes pre-training, instruction tuning, and learning from human feedback. Pre-training predicts the next token across datasets containing "trillions of tokens" or more, and can involve models with hundreds of billions or even trillions of parameters. Training runs can cost tens to hundreds of millions of dollars, so a failed run is difficult to repeat. Instruction tuning uses prompt-response pairs and masks the prompt, training only on the response. It needs far less compute, typically around 100 GPUs for a few hours or days. Preference methods such as reinforcement learning from human feedback and direct preference optimization compare responses, which is cheaper than asking people to write answers from scratch.
Open models can be adapted for lower-cost, task-specific use
The speaker frames model choice as a balance between performance and cost. Commercial models are easier to start with and tend to have stronger general-purpose performance, while open models can do better on a specific task after fine-tuning. A common workflow is to prototype with a high-end model, then move to a smaller open model such as an 7B model or an 8x7 model once the task is understood. The track also presents CodeStral 22B, a dense Transformer model trained for code, with instruct and fill-in-the-middle modes. It supports code completion and questions about bugs, and the speaker says it outperforms larger open code models while using fewer parameters.
Structured generation constrains model outputs before they become invalid
Remy argues that language models are unreliable interfaces because they can fail to return consistent JSON even after prompting and function calling. Outlines lets developers describe the allowed output with regular expressions, JSON schema, Pydantic models, or grammars. During generation, the library examines the logits for each possible next token and masks tokens that would violate the requested structure. This can turn a long answer about an IP address into just the address, or constrain flight fields such as airport codes to three capital letters. Remy presents an experiment where Mistral 7B Instruct produced valid JSON 17% of the time without constraints and 99.9% with structured generation.
Output constraints can improve speed and task accuracy
Remy says structured generation has little inference overhead and can be faster because the model does not need to generate structural tokens such as brackets and field names. In one example, an unconstrained answer used 50 tokens while the constrained answer used eight. On a function-calling benchmark, Microsoft Phi-3 Medium rose from 86% accuracy without constraints to 96.5% with them, compared with 93.5% for GPT-4 on that benchmark. He also describes a GSM-style experiment where structured generation reached the same accuracy with one example as ordinary generation reached with four. Outlines has since expanded from regular expressions to context-free grammars, semantic constraints, and work on reducing computations that become unnecessary once the output structure is known.
Retrieval models need training for source use, citations, and tool actions
Sandra describes Cohere's Command R and Command R+ as open-weight models built for retrieval-augmented generation and tool use. Retrieval systems have to teach the model where to look, distinguish retrieved text from conversation history, resist focusing only on the beginning of a document, and prefer current external information when its pre-training knowledge is outdated. Cohere trains these behaviors so the model can decide whether retrieval is needed, search, use the returned context, and provide fine-grained citations. The company also open-sourced a chat UI toolkit with conversation history, citations, document upload, configurable model access, retrieval, and tool integrations. Command R+ is presented as having similar performance to GPT-4 Turbo at a lower cost.
Multi-step tool use lets models revise plans after errors
Cohere's multi-step API is designed for tasks where each action depends on the previous one. A model might search for a document, compare it with another document, summarize the comparison, and send the result by email. The developer describes the available tools, their purposes, and their parameters. The model then creates a plan, calls tools, reviews their results, and changes the plan when needed. If an API call returns an error, it can retry with a new plan. Sandra shows a demo that searches for companies, retrieves employee counts, runs Python to create a graph, and adds citations. The API's behavior is described in a large multi-step prompt available on Hugging Face.
Finance models combine curated domain data with continued pre-training
Leo explains Gradient's AI Foundry as a collection of custom language models and workflow components. Its finance work uses a domain-specific model and context-length extension. To build the finance dataset, Gradient collects a large corpus, uses membership-inference techniques to filter documents that the base model has probably already seen, reviews the smaller remaining set, and adds synthetic data for variations in format and representation. The training pipeline then applies continued pre-training for domain knowledge, followed by supervised fine-tuning and preference optimization for behavior. Leo compares pre-training with reading textbooks and alignment with learning how to use the information. Gradient released a finance model based on Llama 2 that performed better on finance benchmarks than comparable models.
Fine-tuning can fail from small token and template mistakes
Daniel presents several Llama 3 issues that can damage fine-tuning. A double BOS token can make the training template differ from the inference template and reduce accuracy. The Llama 3 base model should not use the instruct chat template because some special tokens are untrained and can produce NaNs in gradients. If those tokens must be repaired, their values should be set to the mean of the trained tokens, excluding untrained tokens from the calculation. Padding and end-of-sequence tokens must also differ, or the end token can be masked and generation can continue indefinitely. Daniel recommends checking these settings before training and says Unsloth automates several fixes. He also covers correct target modules, LoRA rank and alpha, gradient accumulation, four-bit loading, and non-blocking gradient offloading.
"What you can do with Outlines is actually taking that regular expression that you would use to extract the answer and use it to guide the model to tell the model this is the structure that the output should follow."17:25
Who should watch
You are deciding whether to use an open model or a commercial API and need practical guidance on cost, fine-tuning, and deployment.
Your application needs reliable JSON, function calls, citations, retrieval, or sequences of tool actions rather than free-form chat.
You are fine-tuning Llama 3, extending context, or merging community models and want to avoid configuration mistakes that can silently damage the result.
