# No More Bad Outputs with Structured Generation Rémi Louf, .txt | AI Engineer World's Fair 2024 | 15:32 Source: https://www.youtube.com/watch?v=aNmfvN6S_n4 Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/no-more-bad-outputs-with-structured-generation Published: 2024-10-14 Tags: inference, open-source, structured-outputs ## TL;DR - Structured generation guides a language model toward a required output structure by masking tokens that would violate it. - Outlines can produce valid JSON and other constrained outputs with negligible inference overhead, while sometimes making generation faster. - Adding structure can improve the accuracy of open models, including function calling and few-shot mathematical reasoning. ## Summary Rémi Louf argues that language models are difficult to use inside software because their outputs cannot always be trusted. A request for flight details may produce malformed JSON, even after careful prompting or function calling. Structured generation addresses this by constraining the model during token generation. Outlines checks each possible next token against a regular expression, JSON schema, or other grammar, then masks tokens that would break the required structure. Louf shows examples with JSON, airport codes, vision models, and mathematical answers. He reports that constrained generation can produce valid JSON far more reliably, add almost no inference overhead after compilation, and reduce the number of tokens generated. He also presents results where structure improves few-shot reasoning and function-calling accuracy for open models. The talk closes with work on context-free grammars, semantic constraints for text-to-SQL, and reducing computation spent on tokens that structure already determines. ## Key ideas ### Language models are hard to trust as software components [00:00](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=0s) Louf starts with flight information extraction from emails. The desired result is a JSON object containing fields such as origin and destination, but even heavy prompting and function calling can produce a JSON decode error. He connects this problem to the way software systems are built: APIs and interfaces let separate components work together because their outputs follow agreed rules. When a language model returns inconsistent output, it cannot be treated as a dependable component in an agent or larger infrastructure. Louf says the current technology for agents is therefore not yet sufficient. ### Outlines fits into an existing Python workflow [01:38](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=98s) Outlines is presented as a Python library rather than a framework that dictates how an application must be built. It focuses on open source models and integrates with Transformers, llama.cpp, MLX, and other model providers. Louf says vLLM and TGI have adopted it under the hood, including for function calling. He also credits the project's contributors, whose number was around 87 or 88 at the time of the talk. The basic workflow is to choose a model, create a generator, and call it with a prompt. ### A constraint can be used during generation instead of after it [04:00](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=240s) Without constraints, a model asked for an IP address may generate a long explanation, forcing the application to search the response with a regular expression. Outlines can use that regular expression to guide generation directly. The model then returns only the matching answer. Louf extends the same idea to JSON schemas and Pydantic models. Fields can have detailed types, such as airport codes that must contain three uppercase letters. The library can also apply structured generation to vision models that receive an image and return JSON. ### Structured generation masks invalid next tokens [06:19](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=379s) Louf explains that a model produces a probability distribution over the next token, rather than text directly. Sampling methods such as temperature and top-p already modify those probabilities. Outlines adds another modification: for every possible token, it checks whether adding that token would violate the requested structure. Tokens that would break the regular expression, schema, or grammar are masked. The basic idea is simple. The difficult engineering problem is performing these checks efficiently during inference, which Louf identifies as Outlines' distinction from libraries such as Guidance and LMQL. ### Constraints can make generation both valid and faster [07:23](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=443s) Louf says structured generation is useful because much of the text produced in applications already has a defined form. JSON is one example, while the GSM dataset contains repeated mathematical structure that can be described with a context-free grammar. In one reported experiment, Mistral 7B Instruct produced valid JSON 17.7 percent of the time without structural guidance and 99.9 percent with it. He says Outlines adds negligible inference overhead after compilation. It can also skip tokens whose values are already known, such as JSON punctuation and field names, and prevent a model from producing unnecessary explanatory text. ### Structure can reduce the number of examples needed [11:00](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=660s) Louf describes an accuracy experiment on GSM8K using Mistral 7B Instruct. Without structure, one-shot performance was worse than performance with eight examples, which is expected. With structure, one example reached roughly the same accuracy range as eight examples. His interpretation is that examples may teach the model about the required structure as much as they teach it about the task itself. He calls the result surprising and says the team was continuing to investigate it. ### Constraints raise the function-calling accuracy of smaller open models [11:59](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=719s) On a function-calling benchmark, Louf compares Microsoft Phi-3-medium with and without structured generation. He reports 86 percent accuracy without the constraint and 96.5 percent with it. GPT-4's result on the same benchmark was 93.5 percent. Louf uses this comparison to argue that open models can become useful for practical workflows without fine-tuning when their output format is constrained. The constraint does not make a model generally correct, but it can prevent failures caused by invalid function-call structure. ### The next step is constraining meaning and computation [13:13](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=793s) Louf describes work beyond regular expressions and JSON. Outlines can use context-free grammars for code, protein structures, and mathematical answers. The team is also adding semantic constraints, with text-to-SQL as a practical example. Models often invent table or column names even when their SQL syntax is valid. Louf says their method can guarantee that the generated query will run, although it cannot guarantee that the query returns the intended answer. He also describes efforts to move computation into the model architecture so the model does not spend effort calculating logits for tokens that the structure rules out. ## Notable quotes - "Computing rests on interfaces." (00:56) - "What you get as an output is not text, it's a probability distribution over the next token." (06:39) - "During inference it doesn't slow down inference." (09:39) - "I can guarantee you that it will run." (13:57) ## Tools & references mentioned - Outlines - .txt - OpenAI - Transformers - llama.cpp - MLX - vLLM - TGI - Guidance - LMQL - Mistral 7B Instruct - GSM8K - Microsoft Phi-3-medium - GPT-4 ## Who should watch - You are extracting JSON or function calls from model responses and still handle malformed output with retries, parsing, or prompt tricks. - Your application uses open source models and you want constrained decoding without rebuilding the whole workflow around a new framework. - You are working on text-to-SQL, code generation, mathematical answers, or vision outputs where the allowed structure can be described formally. ## Related talks - [Pydantic is all you need](https://aietalks.com/talks/pydantic-is-all-you-need) (Jason Liu, Independent consultant, 17:55) - [Open Models track](https://aietalks.com/talks/open-models-track-r0x7mpag) (Remy, txt & Sandra, Cohere & Leo, Gradient & Daniel, Unsloth & Maxim Labonne, Liquid AI, 3:34:55) - [Respect The Process](https://aietalks.com/talks/respect-the-process) (Andrew Dumit, Watershed Technology Inc., 16:43) - [Rethinking how we Scaffold AI Agents](https://aietalks.com/talks/rethinking-how-we-scaffold-ai-agents) (Rahul Sengottuvelu, Ramp, 16:32) - [Structuring the Unstructured](https://aietalks.com/talks/structuring-the-unstructured) (Cedric Clyburn, Red Hat, 20:41)