Skills Rulebase
Rulebase QA coverage audit
Audits a Rulebase workspace for the two failure modes that make a QA programme look healthy while producing nothing usable: coverage blind spots (whole segments never evaluated) and instrument failure (scores that don't discriminate or predict anything).
npx rulebase-skills install rulebase-qa-coverage-auditWhen to use it
Reach for this when someone says any of these — they are the phrases the skill itself triggers on:
- “audit our QA coverage”
- “which agents or channels aren't being evaluated”
- “are our QA scores meaningful”
- “is our scorecard working”
How it works
The method, in the order the skill runs it. The full procedure — tables, worked examples and the edge cases — is in the skill itself.
Step 0: Introspect before querying
Do not assume tool signatures or field names. They change between workspace versions and differ by plan. Start every run by discovering the actual shape: get_current_organization, get_workspace_schema, list_scorecards.
Step 1: Coverage
The question is not "how many evaluations did we do" but "what did we never look at".
Step 2: Statistical power per agent
Coverage percentages hide the problem that matters for anyone using these scores to coach or rank. For each agent and period, get the evaluation count n, then report the confidence interval on their score.
Step 3: Distribution health
Pull the score distribution overall and per scorecard: count, mean, standard deviation, p25, p50, p75.
Step 4: Criterion health
For each criterion on each scorecard, compute the fail rate.
Step 5: Outcome linkage
The test of whether the scorecard measures quality: does it relate to anything the business cares about? Aggregate to agent-period level first — conversation-level correlations are dominated by noise.
Step 6: Coaching loop closure
Coverage is worthless if nothing follows from it. Using list_coaching_sessions and the evaluation records: What share of low-scoring evaluations led to a coaching session?.
Related skills
Free and open source, and vendor-neutral — it reads the conversations from whichever helpdesk you already run. Browse all 149 skills · connect your helpdesk over MCP · source on GitHub
