Monitor the Chat Agent and Set a Cost Alert

Lab overview

Once an agent is in front of users, two questions come up every week: what is it doing, and what is it costing? In this lab, you connect Application Insights to contoso-helpdesk-agent, generate traffic across all three of its knowledge paths, inspect a trace span by span, and review the Monitor dashboard for cost, tokens, tool calls, and errors. You finish in Azure Cost Management with a $25 monthly budget on the resource group that emails an alert at 80 percent of actual spend.

Why you'd do this

Until now you've judged the agent by reading its answers. That stops working the day it's in front of fifty help desk staff, because you won't be in the chat. Tracing is how you see what the agent does on every request without being there: which tool it called, which knowledge source it hit, how long each step took, and how many tokens it burned. Foundry writes that telemetry to Application Insights, the same service Azure uses for web apps, so connecting one to the project is the first thing to do before anyone else gets access.

The three test questions are chosen so each one exercises a different knowledge path, and the trace you open afterward shows the shape of a single request: an Invoke Agent span at the top, an Execute Tool span for the knowledge base lookup, and a Chat span for the model call, each with its own duration. When a user says "the agent is slow," this view tells you whether the time went into retrieval or generation. When an answer is wrong, it tells you whether the right tool was even called. Reading a trace is a skill worth getting comfortable with now, while the agent is simple.

The Monitor dashboard aggregates the same data over time: estimated cost, token usage, runs, tool calls, and error rate. Note the Configure buttons for scheduled evaluations and red teaming; those turn the one-time evaluation from lab 9 into a recurring check that runs against live traffic. The dashboard answers "how is it doing this week," which is the question you'll get asked.

The budget is the part most tutorials skip and most organizations regret skipping. Every answer costs tokens, web search is billed per query, and a popular agent can run up a bill quietly. A $25 monthly budget on the resource group with an alert at 80 percent of actual cost means you hear about it from an email, not from finance. Scoping the budget to the resource group rather than the subscription keeps the alert specific to this project. For certification prep, this is squarely the AI-103 objectives on observability, tracing and token analytics, and managing cost footprints, and the Foundry implementation material in AI-901.

Lab details

• Application Insights: contoso-assistant-insights, connected from the agent's Traces tab
• Test Traffic: password expiration, Microsoft Visio request, latest Windows 11 update
• Trace Inspected: Invoke Agent, Execute Tool (knowledge base), and Chat spans
• Monitor Dashboard: estimated cost, token usage, agent runs, tool calls, error rate
• Budget: contoso-assistant-budget, $25 monthly on rg-contoso-assistant
• Alert: 80% of actual cost to [email protected]

Learning objectives

1. Connect Application Insights to a Foundry project to enable agent tracing.
2. Read a trace to see the agent, tool, and model steps behind a single response.
3. Use the Monitor dashboard to review estimated cost, token usage, tool calls, and error rate.
4. Create an Azure budget with an actual-cost alert scoped to a resource group.

Exam relevance

• AI-901 — Microsoft Azure AI Fundamentals — implement AI solutions by using Microsoft Foundry.
• AI-103 — Developing AI Apps and Agents on Azure — manage, monitor, and secure AI systems; manage quotas, rate limits, and cost footprints; set up observability with tracing and token analytics.

How this lab fits into the project

This is lab 11 of 12 in the Contoso Knowledge Assistant project. Contoso's IT help desk fields the same policy questions all day, so over this series you'll build an AI agent that answers them from the company's own documents, then ground it, secure it, evaluate it, monitor it, and publish it to Teams. This lab adds the operational layer: tracing and a dashboard so you can see what the agent is doing, and a budget alert so you'll know what it costs.

Step-by-step instructions

1. Connect Application Insights:
   • In the Foundry portal, click Build, then click contoso-helpdesk-agent on the Agents page.
   • Click the Traces tab, then Connect.
   • Open the Application Insights resource name dropdown and click Create new resource.
   • Double-click the default name, type contoso-assistant-insights, and click Create.
2. Generate some agent traffic:
   • Click the Playground tab.
   • Type What is Contoso's password expiration policy?, click send, and note the knowledge path used.
   • Click the new chat icon, type How do I request Microsoft Visio?, click send, and note the knowledge path used.
   • Click the new chat icon, type What is the latest Windows 11 feature update, and when was it released?, click send, and note the knowledge path used.
   • Notice that each question used a different knowledge path: file search, the knowledge base, and web search.
3. Inspect a trace:
   • Click the Traces tab and click the first trace result.
   • Note the summary at the top: spans, chat calls, tool calls, total duration, and total tokens.
   • Note the Invoke Agent span, with two steps beneath it.
   • Note the Execute Tool span, the call to the contoso-it-kb knowledge base, and the Chat span, the call to the gpt-5.6-sol model.
   • Note the duration beside each span, then click the X to close the trace.
4. Review the monitoring dashboard:
   • Click the Monitor tab and ensure the time range is set to the last 7 days.
   • In the Operational metrics card, note the Estimated cost and Total token usage.
   • Note the Evaluations, Scheduled evaluations, and Scheduled red teaming cards, each with a Configure button.
   • Note the Agent runs chart.
   • Scroll down and note the Tool calls and agent runs chart and the Error rate chart, then scroll back to the top.
5. Create a budget with an alert:
   • Click the Azure portal browser tab and click rg-contoso-assistant.
   • Expand Cost Management and click Budgets, then + Add.
   • Type contoso-assistant-budget as the name and ensure the Reset period is Monthly.
   • Double-click the default amount, type 25, and click Next.
   • Under Alert conditions, open the Type dropdown and select Actual cost.
   • Type 80 in the % of budget field and [email protected] in the Alert recipients field.
   • Click Create and note the new budget in the list.
   • Click Home in the Azure portal.