A fluent answer can still be wrong.
An evaluation needs to ask more than whether the response sounds professional. A system can retrieve the wrong passage, accurately quote an irrelevant passage, or produce a useful answer from material the user should never have seen.
These are different failures. Testing them separately makes it easier to identify what needs to change.
Question: When must I submit an expense claim?
Source: Within 30 days of completing a trip.
Draft: You have 60 days to submit your claim.
Build a small, representative test set.
Start with questions that reflect actual user tasks. Include straightforward cases, ambiguous questions, missing information, conflicting sources and permission boundaries. Record the supporting source or the reason an answer should not be given.
| Check | Question to ask |
|---|---|
| Retrieval | Did the system find the evidence needed for this task? |
| Groundedness | Does the available context support the answer's claims? |
| Relevance | Does the response address what the user actually asked? |
| Permissions | Was every source and action allowed for this user? |
| Operation | Is latency, cost and failure behaviour acceptable for the workflow? |
Use several kinds of checks.
Code can check a schema, a required field or whether a forbidden operation occurred. Model-based graders can help assess qualitative responses, but they need validation and can make mistakes. Human review remains valuable for representative cases, disagreement and consequential failures.
Compare against a baseline.
Keep the questions and grading criteria stable when comparing a change. Record the model, prompt, retrieval configuration and source version. Otherwise a better-looking result may come from changed test conditions rather than a better system.
Try the failure before you trust the success.
Use the evidence lab to test a supported answer, an unanswerable question and a source that the employee role cannot access. Explain the expected behaviour for each case before running it.
For a real application, add domain-specific cases and decide which failures prevent release. There is no single score that makes every AI system ready for every purpose.
References: Microsoft Learn: RAG evaluators and Anthropic: Demystifying evals for AI agents.
Put it into practice
Create your first evaluation card.
Start with ten representative questions and a baseline system. Include normal requests, ambiguous wording, absent evidence and permission boundaries.
- Write expected behaviour and supporting evidence for each question.
- Score retrieval, answer support and access separately; retain the failure examples.
- Change one part of the system and run the same questions again before accepting the change.
A versioned comparison showing what improved, what regressed and which risks remain.