Skills Quality assurance
QA appeal process
Agents will disagree with scores. Without a defined process, disputes become informal lobbying — team leads rewrite numbers, calibration erodes, and the agents who complain loudest win rather than the agents who were right.
npx rulebase-skills install cx-qa-appeal-processWhen to use it
Reach for this when someone says any of these — they are the phrases the skill itself triggers on:
- “QA appeal process”
- “agents disputing scores”
- “who adjudicates QA disputes”
- “overturn rate too high”
How it works
The method, in the order the skill runs it. The full procedure — tables, worked examples and the edge cases — is in the skill itself.
Step 1: Separate contest types
Route differently.
Step 2: Define eligibility and limits
Publish clearly: Window, Scope, Frequency cap, Grounds required.
Step 3: Timeboxes
During open appeal: Do not coach from the contested evaluation, Auto-fail hard tier.
Step 4: Who adjudicates
First level: senior QA analyst or calibrated lead grader — not the original grader, not the agent's team lead.
Step 5: Adjudication standards
Decision options (constrained): Uphold, Overturn, Partial overturn, Remand.
Step 6: Second-level consistency
Track: Overturn rate at L1 vs L2, Overturn direction, Inter-adjudicator agreement on sample of appeals, Criterion-level overturn rate.
Step 7: Close the loop
On every overturn: Update evaluation record with appeal id, adjudicator, reason code, Notify original grader, Feed calibration, Agent communication.
Step 8: Metrics that do not lie
Never rank agents by appeal count. Dispute rate rises when criteria are ambiguous — often the best agents notice first.
Related skills
Free and open source, and vendor-neutral — it reads the conversations from whichever helpdesk you already run. Browse all 149 skills · connect your helpdesk over MCP · source on GitHub
