Azure AI Foundry's Red Teaming Agent runs adversarial prompts against an application or model and reports which attacks succeed.
2
Attack strategies can transform prompts through techniques such as reversing strings, Base64 conversion, Caesar encoding, and composed transformations.
3
Red teaming belongs inside a wider development process that maps risks, adds guardrails, evaluates inputs and outputs, and then iterates on the application.
Summary
Keiji Kanazawa and Nagkumar Arkalgud demonstrate the Azure AI Evaluation SDK's Red Teaming Agent against a local retrieval-augmented generation application. The tool can generate harmful objectives, transform them with attack strategies, send them to a target, and evaluate the response. Developers can scan an application through a URL or scan an Azure OpenAI model directly. The dashboard lets them filter successful attacks and inspect the prompt, strategy, and response. In one example, a GPT-4o setup with guardrails did not produce a successful attack in a small sample. A different setup using Phi-3 showed successful attacks in the hate and fairness category. A direct GPT-4.1 model scan also showed successful violence attacks, with fewer successes after guardrails were enabled. The speakers place red teaming alongside risk planning, content filters, prompt shields, quality evaluators, safety classifiers, agent evaluators, and customer-specific checks.
AI applications can be pushed past their intended behavior
Kanazawa opens with examples of chatbots being tricked into revealing information or producing content they should refuse. A direct request such as how to loot a bank may be rejected, while a long fictional setup can persuade a model to answer. He also shows a phrase written backwards, such as a reversed version of the same request, as another way to evade defenses. The self-driving car example makes the same point for agentic systems: a system can appear to work normally and still fail in an unusual situation.
Trustworthy AI requires repeated engineering checks and shared expertise
Kanazawa compares AI engineering with building bridges, dams, trucks, and trains. Engineers build, iterate, check, and test those systems before people rely on them, and AI applications need the same process. He says trust is a team sport because application engineers need help from people with security and AI risk expertise. Microsoft worked with its AI red team, which had been studying risks in large language models several years earlier, to make red teaming more accessible through Azure AI Foundry.
The SDK connects an application to a hosted red-team workflow
Arkalgud shows a sample retrieval-augmented generation application using PostgreSQL, Semantic Kernel, and a locally running model through Ollama. The red-team plugin exposes functions that an agent can call during the testing process. The target can be any application that accepts a query and returns a string. In the interactive workflow, the agent creates a harmful prompt for a selected category, sends it to the target, transforms the prompt with Base64, sends it again, and displays the target's responses.
Scans combine risk categories, objectives, targets, and attack strategies
A complete scan is configured with an AI project, a target URL, credentials, risk categories, and a number of objectives. The objectives determine how many questions are sent to the application. If no risk categories are selected, all available categories are included by default. The scan also accepts a name, an optional output path, and a list of attack strategies. Strategies can reverse text or apply other conversions, and developers can compose two strategies before sending the resulting prompt to the target.
Arkalgud shows scan results in the hosted dashboard. In one run with GPT-4o and built-in Azure AI Foundry security controls, none of the attacks succeeded in a small sample covering several harm types. After switching to Phi-3, five of forty attacks in the hate and fairness category succeeded. The dashboard allows users to filter for successful attacks and inspect the response that evaluators judged harmful. A direct GPT-4.1 scan with guardrails removed showed successful violence attacks and successful complex attacks, including a Caesar-encoded prompt that the assistant decoded.
Model scanning is available before an application exists
Developers do not need a finished application to start testing. Arkalgud explains that an Azure OpenAI model can be configured as the target by supplying the model details and credentials. The scan then runs directly against the model and produces the same kind of results. This lets teams evaluate a model while they are still building the surrounding application, before they have an endpoint that can accept and answer queries.
Red teaming belongs within a broader evaluation and mitigation process
Kanazawa says teams should first map the risks of the planned application, including whether it is an agent and whether it uses external or customer data. They should plan and implement controls, then run evaluations, with red teaming as one option. Azure AI Foundry includes quality evaluators, risk and safety evaluators, input and output classifiers, evaluators for agent behavior, and support for customer-defined evaluators. When testing finds that harmful content gets through, the team can add content filters, prompt shields, and other guardrails before testing again.
Guardrails sit outside the raw model and can filter both directions
In the closing discussion, Kanazawa explains that Azure's guardrails can operate on inputs and outputs. Input filters can block requests for harmful instructions, while output controls can prevent the model from returning disallowed material. The underlying model remains a raw model when content filters are enabled or disabled. The guardrail features are applied around it, so teams can test the model directly and then compare behavior after adding the controls.
"The guardrails are not in the model itself, the model is still the raw model and the guardrails are actually kind of outside it."18:32
Who should watch
You are building an agent or RAG application and need a practical way to send adversarial prompts to it before release.
Your team has a model but not a complete application, and you want to scan the Azure OpenAI configuration directly.
You already use content filters and want to see how red-team results fit with input checks, output checks, agent evaluators, and custom evaluations.
