Skills Quality assurance
Evaluating an AI support agent
Support AI is usually shipped on vibes: someone tries ten questions, it looks good, it goes live. Then a prompt tweak three weeks later breaks a category nobody re-tested, and the first signal is a complaint.
npx rulebase-skills install cx-ai-agent-evaluationWhen to use it
Reach for this when someone says any of these — they are the phrases the skill itself triggers on:
- “how do we test our support bot”
- “did the new prompt make it better”
- “evaluate our AI agent”
How it works
The method, in the order the skill runs it. The full procedure — tables, worked examples and the edge cases — is in the skill itself.
Build the eval set from real traffic
Not from imagined questions. Real customers ask things nobody would think to write.
Define what a correct answer is, per case
For each case, record the expected outcome — not the expected wording. Wording-match scoring fails a correct answer phrased differently and is the most common reason teams abandon eval sets.
Grade on dimensions that fail independently
A single quality score hides the failure that matters. Grade at least: Completeness, Grounding, Appropriate deferral, Safety, Tone, against the documented standard. Last, and never a gate.
Grading at a workable cost
Hand-grading is the gold standard and does not scale to every change. The workable compromise: Hand-grade the initial run to establish truth for the set, Re-validate the grader whenever the grader's own model changes, Hand-grade a sample of every run anyway, to catch grader drift.
Regression, not just evaluation
The eval set earns its keep on the second and subsequent runs.
Offline evaluation is necessary and not sufficient
An eval set measures answers to fixed questions. It does not measure: Multi-turn behaviour. Most real failures happen in turn three, after a, Retrieval on the live corpus as it drifts, Real customer phrasing, which changes, Whether customers were satisfied, which needs production signal.
Related skills
Free and open source, and vendor-neutral — it reads the conversations from whichever helpdesk you already run. Browse all 149 skills · connect your helpdesk over MCP · source on GitHub
