Skills Quality assurance
Auditing a rubric for false positives
The request usually arrives as "the scores keep dropping, there are too many false positives, how do we make it more lenient". Underneath it is a real and serious problem — a QA programme the team has stopped believing — and a proposed remedy that would make it worse.
npx rulebase-skills install cx-rubric-false-positive-auditWhen to use it
Reach for this when someone says any of these — they are the phrases the skill itself triggers on:
- “our evaluation scores keep tanking”
- “there are too many false positives”
- “how can we be more lenient”
- “what exceptions do I need to add to the scorecard”
How it works
The method, in the order the skill runs it. The full procedure — tables, worked examples and the edge cases — is in the skill itself.
Step 1: assemble the candidate set, from two places
The disputed evaluations are the obvious source and they are selection-biased: they are the ones someone had the energy to contest. Use them, but not alone.
Step 2: adjudicate against policy, not against feeling
For each candidate, decide whether the flag was correct — by reference to documented policy and the rubric's own stated test, not by whether the outcome feels harsh.
Step 3: classify the cause
This is the analytical core. Every false positive has one of five causes, and they have five different fixes. Lumping them together is what produces the "just be more lenient" request in the first place.
Step 4: check for the causes that look like harshness but are not
Before recommending any rubric change, rule out three things that produce a falling score with no change in either the rubric or the team.
Step 5: quantify each fix before making it
For every proposed change, estimate the effect on a frozen historical sample before shipping: How many evaluations in the window would have changed verdict?, What would the aggregate score have been?.
Step 6: version and announce
Any rubric change breaks comparability. Bump the scorecard version, record the date, and state plainly which historical periods are no longer comparable. A quiet edit turns the next quarter's trend into fiction, and the trust problem you were hired to fix comes back worse — now with the added complaint that the numbers change without notice.
What "more lenient" must never mean
If someone insists on a global relaxation after seeing the analysis, that is their call to make — say clearly what it costs (false negatives, a broken series, no diagnosis of the actual causes), record that the recommendation differed, and implement it as a versioned, announced change rather than a quiet one.
Related skills
Free and open source, and vendor-neutral — it reads the conversations from whichever helpdesk you already run. Browse all 149 skills · connect your helpdesk over MCP · source on GitHub
