Skills Quality assurance
Grader agreement and calibration
Two graders can disagree in two completely different ways, and the fixes are opposites: Noise, Bias.
npx rulebase-skills install cx-calibration-agreementWhen to use it
Reach for this when someone says any of these — they are the phrases the skill itself triggers on:
- “run a calibration”
- “how do our reviewers compare”
- “is the AI grading too harshly”
- “our QA scores have too many false positives”
- “why do reviewers disagree”
How it works
The method, in the order the skill runs it. The full procedure — tables, worked examples and the edge cases — is in the skill itself.
First, which question is this?
Calibration — graders independently score a designed gold set. Clean, causal, comparable over time, and the only way to measure a criterion. This is the measurement instrument.
Never use raw percentage agreement
With a 95% pass rate, two graders who assign verdicts at random and independently agree about 90% of the time. "We have 90% agreement" is therefore consistent with having learned nothing at all.
Separating bias from noise
This is the step that changes what you do next, and it is usually skipped.
Differential agreement is a fairness issue
Compute agreement per segment — channel, language, market, team, tenure — not just overall. If an AI grader agrees with humans 88% of the time on email and 62% on voice, the aggregate figure of 84% hides a system that is unreliable on voice, and every voice agent is being scored by a worse instrument than their email colleagues.
Contested and overturned evaluations
Overturn rate looks like an obvious quality signal and is one of the most misleading numbers available, because only disputed evaluations get re-reviewed. A high overturn rate among contested items is expected: people contest what they think is wrong. It says nothing about the base rate.
Related skills
Free and open source, and vendor-neutral — it reads the conversations from whichever helpdesk you already run. Browse all 149 skills · connect your helpdesk over MCP · source on GitHub
