Skills Quality assurance
Bot knowledge grounding audit
A fluent wrong answer is usually two failures stacked: retrieval did not surface the right source, and generation asserted anyway. Teams tune the model when the KB is empty, or expand the KB when search returns the wrong chunk. Measure retrieval and generation separately or you will fix the wrong layer.
npx rulebase-skills install cx-bot-knowledge-groundingWhen to use it
Reach for this when someone says any of these — they are the phrases the skill itself triggers on:
- “why does the bot make things up”
- “RAG audit”
- “wrong article cited”
- “retrieval quality”
- “bot answers not in the KB”
- “grounding failures”
How it works
The method, in the order the skill runs it. The full procedure — tables, worked examples and the edge cases — is in the skill itself.
Failure taxonomy
For each bad or unsupported answer, classify root cause.
Measure retrieval without generation
Run retrieval-only evaluation on a labelled query set: Query, Expected document(s), Metrics.
Measure generation given retrieval
Fix retrieval logs, then re-run or replay: Supported, Unsupported, Contradicted.
Missing KB vs wrong chunk
Signals for missing KB: Repeated questions on the same driver with fabrication or hand-waving.
Stratified sampling for grounding audit
Do not sample uniformly. Over-weight: Regulated, fees, eligibility, timelines, Drivers with high repeat contact after bot resolution, Languages with known thin content, Cases where the bot cited something.
Related skills
Free and open source, and vendor-neutral — it reads the conversations from whichever helpdesk you already run. Browse all 149 skills · connect your helpdesk over MCP · source on GitHub
