Evaluate the Knowledge Assistant Agent

Lab overview

Testing one question at a time doesn't scale. In this lab, you run a formal evaluation against contoso-helpdesk-agent: upload a ten-question JSONL dataset, choose contoso-chat as the judge model, trim the suggested evaluators, and submit the run. Then you read the results, from overall pass rates down to a single row where a failing score turns out to be the agent doing exactly what it was told.

Why you'd do this

Every lab so far has tested the agent the same way: type a question, read the answer, nod. That works for three questions and falls apart at thirty, and it tells you nothing about whether the change you made last week broke something that worked the week before. An evaluation replaces that with a fixed set of questions, scored the same way every time, so you get a number per version instead of an impression. This lab establishes that baseline for version 3; when you change the agent later, you rerun the same dataset and compare.

The scoring is done by another model, a pattern called LLM-as-judge. The judge reads each question, the agent's answer, and the tools it called, and grades them against criteria such as whether the right tool was selected, whether the answer stayed on task, and whether it was relevant. Choosing contoso-chat as the judge keeps the lab simple; in practice you'd often pick a stronger or at least a different model than the one being judged, so the grader isn't blind to its own habits. The JSONL dataset is just one question per line with the fields the evaluators map to, which is why the field mapping step matters: a mismapped column scores everything wrong silently.

Foundry suggests twenty evaluators in three groups. Removing the Safety group is a judgment call you'll make often: those evaluators are valuable, but they're expensive to run and the help desk agent's guardrails from lab 4 already enforce safety at the deployment level, so for a quality baseline they add cost without changing what you learn. The remaining thirteen cover tool use and answer quality, which is what this project is about.

The last step is the lesson. The chili recipe question scores a Fail on customer satisfaction, and that's correct: the agent refused an out-of-scope request, exactly as its instructions say. A failing number isn't automatically a bug. Reading the detailed rows, not just the pass rates, is how you tell a real regression from an agent doing its job. For certification prep, agent evaluation and quality metrics are AI-103 material, and responsible AI evaluation concepts appear in AI-901.

Lab details

• Target: contoso-helpdesk-agent, version 3
• Scope and Frequency: individual turns, one time
• Dataset: contoso-helpdesk-eval (10 questions, JSONL)
• Judge Model: contoso-chat
• Evaluators: 13 (Agents and Quality groups; Safety removed)
• Run Name: contoso-helpdesk-eval-01

Learning objectives

1. Create an evaluation for an agent and upload a JSONL test dataset.
2. Explain LLM-as-judge and choose a judge model.
3. Select evaluators and interpret pass rates for tool use, task adherence, and relevance.
4. Recognize when a low score reflects correct behavior on an out-of-scope question.

Exam relevance

• AI-901 — Microsoft Azure AI Fundamentals — AI agents and their components, grounding with your own data.
• AI-103 — Developing AI Apps and Agents on Azure — create a prompt agent in Foundry Agent Service, agent instructions, tools, and versions.

How this lab fits into the project

This is lab 9 of 12 in the Contoso Knowledge Assistant project. Contoso's IT help desk fields the same policy questions all day, so over this series you'll build an AI agent that answers them from the company's own documents, then ground it, secure it, evaluate it, monitor it, and publish it to Teams. This lab establishes a baseline for version 3 of the agent, so every later change can be measured against the same ten questions.

Step-by-step instructions

1. Start a new evaluation:
   • In the Foundry portal, click Build, then click contoso-helpdesk-agent on the Agents page.
   • Click the Evaluation tab, then Create.
2. Choose the target, scope, and frequency:
   • Ensure Agent is selected as the evaluation target and click Next.
   • Ensure Individual turns is selected as the scope and click Next.
   • Ensure One time is selected as the frequency and click Next.
3. Add the test dataset:
   • Select Existing dataset as the dataset source, then click Upload new dataset.
   • Click Choose file, select contoso-helpdesk-eval.jsonl, and click Open.
   • Click the New dataset name field, type contoso-helpdesk-eval, and click Upload.
   • Click Next.
   • On Configure agents, note that the evaluation targets contoso-helpdesk-agent version 3, and click Next.
4. Select the testing criteria:
   • Note that contoso-chat is selected as the judge model.
   • Scroll down and note that 20 evaluators were suggested automatically, in three groups: Agents, Quality, and Safety.
   • Note the mapped fields on the right: query, response, tool_calls, and tool_definitions.
   • Click Remove all next to Safety, then click Next.
5. Review and submit:
   • Double-click the default evaluation name and type contoso-helpdesk-eval-01.
   • Review the target, dataset, and evaluators, scrolling down to see the full list.
   • Click Submit.
   • Wait for the status to change from In Progress to Completed.
6. Read the results:
   • Click contoso-helpdesk-agent to open the completed run.
   • Note the Status of Completed, the Duration, and that the Agent ID shows version 3.
   • Under Overall metric results, note the pass rate for ToolSelection, TaskAdherence, and Relevance, then scroll over to view the remaining evaluators.
   • Under Detailed metrics result, find the row for What's a good chili recipe? and note that CustomerSatisfaction shows Fail.
   • Click Microsoft Foundry in the upper left to return to the home page.