The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral

Pamela Fox, Microsoft51:38 · Oct 2026 · 7,740 views
Thumbnail for The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral Watch on YouTube
TL;DR
  1. 1

    A model benchmark does not predict how a model will behave in a specific application, so engineers should test models against their own scenarios.

  2. 2

    Prompt engineering can fix some weaknesses, such as letter counting and tool-call arguments, but it cannot make a model understand images that it cannot interpret.

  3. 3

    Automated evaluations should combine exact checks, LLM judges, generated test sets, and prompt optimization when teams need to compare models or move to a cheaper one.

Summary

Pamela Fox runs the same application scenarios against GPT-5.5, Kimi K2.6, DeepSeek V4 Flash, and Mistral Large 3 through Microsoft Foundry. The tests cover letter counting, RAG grounding, image understanding, multilingual input, tool calls, tool-call loops, parallel calls, and a LangChain travel agent. The models differ in ways that standard benchmarks would not reveal. Some prompt changes improve counting, grounding, and tool arguments, while image understanding depends more directly on the model's visual ability. Fox then builds evaluations, starting with string checks and moving to an LLM judge that inspects outputs and traces. Microsoft's ASSERT tool generates scenarios and judging rubrics from a YAML specification. Finally, DSPy searches for a prompt that lets Mistral satisfy a set of constraints, offering a way to use a cheaper model without hand-tuning prompts for every swap.

Key ideas
04:58

Application scenarios reveal differences that benchmarks miss

Fox says benchmarks can give an approximate view of quality, safety, cost, and throughput, but they do not show how a model will behave in a particular application. Her workshop therefore runs the same code against multiple models. The examples begin with a simple LLM call and progress to RAG, images, tool calls, and agents. She frames model portability as an engineering requirement because new models appear often and access to a model can change. The practical question is whether an application still works after changing the model, or what must be changed to make it work.

05:38

Simple prompts expose sharp differences in basic reasoning

The first test asks how many times the letter E appears in the sentence, "The elderly gentleman eagerly entered the elevator." GPT-3.5 and Kimi return 13, while DeepSeek V4 Flash returns 6 and Mistral returns 8. Fox then changes the prompt to ask the model to split the sentence into words, count each word, sum the results, and show its reasoning. With that prompt, Mistral gets the answer right. The example shows that prompt changes can close some gaps, while also making the prompt itself part of the model comparison.

08:38

RAG quality depends on whether a model admits what is absent

Fox tests a carpenter-bee dataset with a question about both carpenter bees and honey bees. Since the sources contain only carpenter-bee information, a good answer should explain the carpenter-bee facts and say that the sources do not cover honey bees. GPT-5.5 and Kimi handle this well, and DeepSeek also says the sources lack the requested information. Mistral answers from its own knowledge instead. A stronger instruction gets Mistral to label the unsupported details as having no source, but it still includes them. Fox says teams should evaluate how willing a model is to admit what it does not know.

12:58

Image understanding and language can change the result completely

The image tests include an aurochs, crocodiles, and a screenshot of a plant-price table. GPT-5.5 identifies the images and finds the cheapest plant, while Kimi also performs well. DeepSeek cannot be tested because it accepts text only. Mistral knows the first image is not a unicorn, but fails to identify the aurochs, calls the crocodiles alligators, and selects a plant priced at $2.30 instead of the 58-cent option. Fox says prompt engineering cannot generally repair an image-understanding failure. She also finds that switching the crocodile question to Spanish causes a model to confuse related terms.

17:13

Tool schemas are prompts that shape how models call functions

Fox describes tool calling as a model choosing a function and producing its arguments, while the application actually executes the function. She defines a calendar function with JSON Schema, including the function name, descriptions, parameter names, and types. GPT-5.5 adds Fox as an attendee even though she was not requested. DeepSeek includes the user as an attendee and formats the virtual location as "virtual (Microsoft Teams)." These failures expose ambiguity in the schema and descriptions. Fox recommends making constraints explicit, such as saying that only named attendees should be included.

21:00

Agent evaluation includes efficiency and the whole tool-call trace

In a calculator loop, GPT-5.5 solves a word problem with one tool call by sending one complete expression. DeepSeek reaches the correct result through four calls, which adds network requests and time. Fox then uses a LangChain v1 travel planner with tools for flights, hotels, budgets, and activities. GPT-5.5 calls flight and hotel searches in parallel, then checks the budget and searches for activities. Kimi takes extra turns and does not parallelize. Mistral stops after searching for flights and hotels. Fox says agent evaluation must inspect the tools selected, the arguments, the number of calls, parallelism, and whether the agent completes all required steps.

30:27

Exact checks are useful before an evaluation needs an LLM judge

Fox starts with direct checks for tool calls. A test can compare the selected function and its arguments against expected values, including whether the location is exactly "virtual." She recommends maintaining many test cases and adding new cases from real production inputs so the evaluations do not drift away from actual usage. For outputs that are too flexible for string comparison, she uses an LLM as a judge. The judge receives the query, response, and relevant context, then returns pass or fail with a reason. For agent evaluation, it may also need tool definitions and the complete trace.

40:05

ASSERT generates evaluation cases and rubrics from a specification

Fox presents Microsoft's new ASSERT tool, which takes a YAML specification describing an agent, its users, and the behavior that matters. ASSERT expands the specification into judging rubrics, behavior categories, and generated scenarios. Her travel-planner example includes budget pressure, direct-flight preferences, and required tool use. The generated test set helps teams bootstrap evaluations before they have enough production data. In Fox's comparison, GPT-5.5 has fewer violations than Kimi and DeepSeek for this travel-planner scenario. She cautions that custom evaluators are more useful than generic checks such as coherence and fluency.

45:50

DSPy searches for a prompt that a cheaper model can satisfy

Fox uses DSPy to declare a program's constraints instead of writing its prompt directly. The example requires a numbered list with exact word counts and first letters that spell a target word. DSPy evaluates candidate prompts and uses a genetic optimization process to search for one that satisfies the checks. Fox runs the optimizer against Mistral, the model that had struggled most in earlier examples. The resulting system prompt works with Mistral, showing how a team could optimize prompts for a cheaper model rather than manually rewriting instructions after every model swap. She notes that this approach changes how the application is structured because the prompt becomes an output of the optimization process.

"You don't really know how good a model is until you use it for your particular scenario because there's so many different ways of using models."04:58
Who should watch
  • You are choosing a model for a RAG, vision, tool-calling, or agent application and need tests that reflect its actual inputs.
  • You want to compare models on cost or availability without assuming that a benchmark ranking will carry over to your product.
  • Your team has prompts and evals already, but needs generated test cases or a way to optimize prompts for a smaller model.