Validating an AI grader against humans

When to use it

Reach for this when someone says any of these — they are the phrases the skill itself triggers on:

• “can we trust the AI grader”
• “how accurate is our automated QA”
• “validate the grader before rollout”
• “AI vs human agreement by channel or language”
• “is the grader too harsh on chat”

How it works

The method, in the order the skill runs it. The full procedure — tables, worked examples and the edge cases — is in the skill itself.

1. Separate calibration from production agreement

   Calibration — the same fixed set of conversations, graded independently by humans and by the AI grader. This is the only fair comparison when you change the grader's model or prompt.

2. Measure agreement by segment, not only in aggregate

   Overall agreement is a headline, not a decision. Slice at least.

3. Diagnose noise versus bias

   Two graders can disagree in opposite ways: Noise, Bias.

4. Choose thresholds from your costs, not from folklore

   There is no universal "good enough" agreement. Derive a threshold from what happens when the grader is wrong.

5. Triage versus replace

   Never promote from triage to replace on aggregate agreement alone. Require segment tables, a false-pass audit on stratified samples, and a plan for appeals.

Related skills

• AI agent evaluationTest an AI agent before shipping a prompt or model change
• Bot safety auditFind where a bot did harm, and where it was pushed to
• Copilot evaluationMeasure whether the copilot actually helps agents
• Limits on AI actionsLimit what an AI agent may do to a customer’s record
• What the bot can findFix what the bot retrieves before blaming what it says
• Where humans approveDecide where a human has to approve, and what that costs