Skills Quality assurance
Evaluating agent assist (copilot)
High acceptance rate is the metric vendors love and operators regret. An agent clicks "accept" because it is faster than rewriting, because the suggestion is good, or because they stopped reading. Acceptance measures friction, not value.
npx rulebase-skills install cx-agent-assist-evaluationWhen to use it
Reach for this when someone says any of these — they are the phrases the skill itself triggers on:
- “is our copilot working”
- “measure agent assist ROI”
- “acceptance rate is high but is it good”
- “did the draft suggestion help”
- “evaluate our support copilot”
How it works
The method, in the order the skill runs it. The full procedure — tables, worked examples and the edge cases — is in the skill itself.
Define the unit of observation
Pick one primary unit and stick to it: Suggestion event, Reply sent, Conversation.
Measure value beyond acceptance
Edit distance is the workhorse: character- or token-level distance between suggestion and sent reply, bucketed (sent verbatim / light edit / heavy rewrite / discarded). A copilot that is accepted then heavily edited is not delivering value — it is delivering a rough draft the agent distrusts.
Quality must move with efficiency
Efficiency gains that increase policy errors or omit required steps are negative ROI.
Acceptance of bad suggestions
Track explicitly: Bad suggestion offered, Bad suggestion accepted.
Experiment design without fake precision
Randomised A/B — best when you can assign agents or conversations cleanly and power is honest. State primary metric upfront (e.g. median time to send on matched drivers). Report confidence intervals; if n is small, say the experiment is directional only.
Related skills
Free and open source, and vendor-neutral — it reads the conversations from whichever helpdesk you already run. Browse all 149 skills · connect your helpdesk over MCP · source on GitHub
