Ground the Chat Model on Policy Documents

Lab overview

Give contoso-chat knowledge it doesn't have. In this lab, you remove the default Web search tool, capture an ungrounded baseline, upload four Contoso IT policy documents into a vector index, and attach them with file search. Then you ask the same questions again and watch the answers change, complete with citations, and see how the model handles a question the documents don't cover.

Why you'd do this

A chat model knows what it was trained on, which is the public internet up to a point in time and nothing about your company. Grounding fixes that without retraining: you hand the model your own documents and tell it to look there first. The pattern is called retrieval-augmented generation (RAG), and it's the standard way to build an assistant that answers from a policy manual, a product catalog, or a ticket history. Every "chat with your data" product you've seen works this way.

The vector index is the part that makes retrieval work. When you upload the four policy documents, Foundry splits them into chunks and converts each chunk into a numeric embedding that captures its meaning. When a user asks about password expiration, the question gets the same treatment, the index finds the chunks closest to it, and those chunks are fed to the model alongside the question. That's why the grounded answer costs more tokens than the baseline: the model is reading the relevant policy text on every call. It's also why the answer comes with a citation. The model isn't remembering, it's quoting.

Two things in this lab are worth watching for. Removing Web search first is deliberate: with it attached, the model could answer from a random blog post and you'd never know whether grounding was working. And the vacation-days question at the end is the real test of a knowledge assistant. A grounded model that says "the documents don't cover that" is behaving correctly; one that invents an answer is the failure mode you're trying to prevent. Lab 4's guardrails stop harmful output; grounding stops confident nonsense.

For certification prep, this is the grounding-with-your-own-data objective in AI-901 and the tools and knowledge sources material for agents in AI-103.

Lab details

• Deployment: contoso-chat
• Vector Index: contoso-policies
• Documents: Device Replacement, VPN Access, MFA Enrollment, and Password and Account Lockout policies
• Tool: File search (Web search removed)
• Test Prompts: password expiration, MFA setup, VPN troubleshooting, vacation days

Learning objectives

1. Explain retrieval-augmented generation and the role of a vector index.
2. Upload documents and attach them to a model with file search.
3. Compare grounded and ungrounded answers to the same question.
4. Read citations and token counts to verify where an answer came from.

Exam relevance

• AI-901 — Microsoft Azure AI Fundamentals — AI agents and their components, grounding with your own data.
• AI-103 — Developing AI Apps and Agents on Azure — create a prompt agent in Foundry Agent Service, agent instructions, tools, and versions.

How this lab fits into the project

This is lab 5 of 12 in the Contoso Knowledge Assistant project. Contoso's IT help desk fields the same policy questions all day, so over this series you'll build an AI agent that answers them from the company's own documents, then ground it, secure it, evaluate it, monitor it, and publish it to Teams. This lab closes the gap lab 3 exposed: the model finally answers from Contoso's own policy documents.

Step-by-step instructions

1. Open the playground and remove Web search:
   • In the Foundry portal, click Build, then Models.
   • Click contoso-chat to open its playground.
   • In the Tools section, click the ⋮ next to Web search and click Remove.
2. Get an ungrounded baseline:
   • In the chat field, type What is Contoso's password expiration policy? and click send.
   • Note that the model doesn't know Contoso's policy.
   • Note the token count shown under the answer.
3. Attach the policy documents:
   • Click Upload files and note that a new index is created.
   • Double-click the default vector index name and type contoso-policies.
   • Click browse for files.
   • Click Contoso-Device-Replacement-Policy, then hold Shift and click Contoso-Password-and-Account-Lockout-Policy to select all four documents.
   • Click Open and wait for the uploads to complete.
   • Click Attach.
   • Note that File search now appears in the Tools section.
4. Ask the same question again:
   • In the chat box, type What is Contoso's password expiration policy? and click send.
   • Note the answer: passwords expire every 120 days, with reminders 14, 7, and 1 day before.
   • Note the citation to the Password and Account Lockout Policy document, and the higher token count compared to the baseline.
5. Test the other documents:
   • Type How do I setup multifactor authentication on a new phone? and click send.
   • Note the Contoso-specific steps, including keeping the old phone for 48 hours.
   • Type A user can sign into email but not the VPN, and their password was changed yesterday. What should they do? and click send.
   • Note that the answer now points to signing out of Contoso SecureConnect and back in.
   • Note the citation to the VPN and Remote Access Policy document.
6. Ask about something the documents don't cover:
   • Click the new chat icon.
   • Type How many vacation days do Contoso employees get? and click send.
   • Note that the model says the documents don't cover it.
   • Click Microsoft Foundry to return to the home page.
   • Click Leave without saving when prompted to save as an agent.