# Creating and scaling your own custom copilots with Azure AI Studio

Hanchi Wang, Microsoft | AI Engineer World's Fair 2024 | 24:21

Source: https://www.youtube.com/watch?v=NjNEdjDvKu8
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/creating-and-scaling-your-own-custom-copilots-with-azure-ai-studio
Published: 2025-02-06
Tags: copilots, evals, observability, tool-use

## TL;DR
- AI applications need domain knowledge, tools, safety filtering, evaluation, and monitoring rather than only an LLM and retrieval.
- Prompt Flow adds tracing, local testing, evaluation, and monitoring workflows to Azure AI Studio applications.
- Evaluation on a custom dataset can compare models across quality, speed, errors, safety, and SQL similarity before deployment.

## Summary
Hanchi Wang explains how Azure AI Studio and Prompt Flow support the full development cycle for custom copilots. He uses a chatbot for an outdoor equipment company as the example. The chatbot turns natural-language sales questions into SQL, fetches data, and uses a code interpreter to draw charts. Prompt Flow instruments the application so developers can inspect inputs, outputs, generated SQL, and tool calls. It also supports local testing, evaluation datasets, content safety checks, custom evaluators, and model comparisons. Wang compares models using execution time, error rate, and SQL similarity, then shows how OpenTelemetry traces can feed an Application Insights dashboard after deployment. The talk is practical about failure: the demo initially produces an incorrect chart, which becomes a reason to inspect traces and run evaluations. Azure AI Studio provides the shared cloud view for traces, assets, and results, while Prompt Flow can also run locally and independently.

## Key ideas
### Custom copilots need engineering around the language model
[01:01](https://www.youtube.com/watch?v=NjNEdjDvKu8&t=61s)
Wang says AI applications cannot simply be attached to existing software as plug-in components. Large language models need access to domain-specific knowledge and tools. Their outputs need content-safety filtering and regular evaluation. After deployment, engineers must monitor performance to see whether the application continues to work as expected. He introduces Azure AI Studio and Prompt Flow as tools for building, testing, tracing, evaluating, deploying, and monitoring these applications. Azure AI Studio includes models, cognitive capabilities, search, machine learning, and tools for safe and secure AI systems. Prompt Flow adds developer tools and can run locally without an Azure account.

### The example copilot combines SQL generation with chart creation
[04:23](https://www.youtube.com/watch?v=NjNEdjDvKu8&t=263s)
The example is a chatbot for an outdoor equipment company that answers sales-data questions. It uses the Assistants API with two tools. One translates a natural-language question into a SQL query and fetches the data. The other is the built-in code interpreter, which uses the data to create charts. The chatbot keeps the conversation history, so a follow-up request can refer to the previous question without repeating it. Wang asks it to overlay a bar chart with a line showing the percentage difference from 2023. The first result is incorrect, which gives him a concrete example of why evaluation and monitoring are needed.

### Tracing exposes each layer of a tool-using application
[08:49](https://www.youtube.com/watch?v=NjNEdjDvKu8&t=529s)
Prompt Flow's trace decorator captures inputs, outputs, and events from Python functions. It can also capture interactions with language models such as Azure OpenAI and OpenAI. Wang traces the chatbot, the sales-data tool, the generated SQL, and the code interpreter's Python function. The local Prompt Flow UI lets developers run a function, watch traces arrive, and drill from the overall chatbot output into individual tool calls. The traces show how the Assistants API changed the question, which SQL query it generated, and how the chart was produced. Prompt Flow emits OpenTelemetry traces and events, so teams can send them to a destination of their choice.

### Azure AI Studio makes traces shareable across a team
[13:47](https://www.youtube.com/watch?v=NjNEdjDvKu8&t=827s)
Prompt Flow can run entirely on a developer's machine, but Azure AI Studio provides a shared view for longer-term storage and collaboration. Wang configures the local application's traces to go to Azure AI Studio, where he sees the same trace details in a browser. A developer can copy the trace URL and share it with a colleague. The tracing approach applies to applications ranging from a single language-model call to retrieval, function calling, and multi-agent workflows. This gives teams a way to inspect the same execution rather than trying to reproduce an issue from a verbal description.

### Evaluation needs a representative dataset and explicit checks
[14:51](https://www.youtube.com/watch?v=NjNEdjDvKu8&t=891s)
Wang describes language-model applications as unpredictable because they can retrieve the wrong data or produce incorrect responses. Prompt Flow's evaluation workflow connects a test dataset, a target application, and evaluators. His dataset contains user questions and the expected SQL generated by the sales-data tool. Prompt Flow can help generate synthetic test data, though he uses a prepared dataset in the demo. The built-in content-safety evaluator checks whether generated queries and outputs contain harmful, hateful, or violent content. Custom Python evaluators measure execution time, error rate, and SQL similarity against the expected query. Evaluation results are available in Azure AI Studio, including aggregate metrics and the traces behind individual results.

### Model comparison depends on the application's tradeoffs
[19:29](https://www.youtube.com/watch?v=NjNEdjDvKu8&t=1169s)
Wang compares several models from the Azure AI Studio model catalog on the same custom dataset. Against GPT-3.5 Turbo as the baseline, Mistral Large has a higher execution time, a better error rate, and a higher SQL similarity score. GPT-4 Turbo has the highest execution time of the three, the same error rate as Mistral Large, and the highest SQL similarity score. For this application, Wang concludes that GPT-4 Turbo is the best choice if speed is not a concern. The example shows why benchmark results alone do not answer every model-selection question. Teams can use their own data and measure the properties that matter to their application.

### Production monitoring connects individual failures to system-wide patterns
[21:30](https://www.youtube.com/watch?v=NjNEdjDvKu8&t=1290s)
Deployment does not finish an engineer's work, according to Wang. Developers need to inspect individual requests as well as telemetry for the whole system. Because Prompt Flow emits OpenTelemetry spans and events, the deployed chatbot can send traces to Application Insights. Wang's dashboard includes performance, usage, and failure metrics. GPT-4 Turbo has the highest model duration, matching the earlier evaluation result. The dashboard also shows that most token usage goes into the prompt rather than the completion. If cost is a concern, Wang says the team may need to shorten its prompt templates. A specific trace can be opened to inspect the same trace view and the stack trace when something fails.

## Notable quotes
- "AI is not just a plug and play solution." (01:01)
- "The code we developers write doesn't always run the way we expect." (08:49)
- "Deployment does not mean the job is done for an engineer, on the contrary, it's the start of a new journey." (21:30)
- "If speed is not a concern then GPT-4 Turbo is still the best choice for my situation." (20:28)

## Tools & references mentioned
- Azure AI Studio
- Prompt Flow
- Assistants API
- OpenTelemetry
- Application Insights
- Azure OpenAI
- OpenAI
- GPT-3.5 Turbo
- GPT-4 Turbo
- Mistral Large
- Llama

## Who should watch
- You are building a copilot that calls tools or generates structured outputs such as SQL, and you need to inspect how each step runs.
- Your team wants local tracing and evaluation during development, with shared trace views and dashboards after deployment.
- You need to compare language models on your own application data instead of relying only on published benchmarks.

## Related talks

- [Agentic Excellence: Mastering AI Agent Evals with Azure AI Evaluation SDK](https://aietalks.com/talks/agentic-excellence-mastering-ai-agent-evals-with-azure-ai-evaluation-sdk) (Cedric Vidal, Microsoft, 20:01)
- [Build, Evaluate and Deploy a RAG-Based Retail Copilot with Azure AI](https://aietalks.com/talks/build-evaluate-and-deploy-a-rag-based-retail-copilot-with-azure-ai) (Cedric Vidal, David Smith & Miguel Martinez, Microsoft, 1:57:58)
- [Running AI Applications in Minutes with AI Templates](https://aietalks.com/talks/running-ai-applications-in-minutes-with-ai-templates) (Gabriela de Queiroz, Pamela Fox & Harald Kirschner, Microsoft, 1:29:02)
- [Mind the Gap (In Your Agent Observability)](https://aietalks.com/talks/mind-the-gap-in-your-agent-observability) (Amy Boyd & Nitya Narasimhan, Microsoft, 1:20:07)
- [Fighting AI with AI](https://aietalks.com/talks/fighting-ai-with-ai) (Lawrence Jones, Incident.io, 17:29)