Most support quality scorecards start the same way: someone copies a template with twenty-five line items, every conversation scores in the high eighties, and a quarter later nobody can say what the numbers changed. Agents see the score as a grade, reviewers see it as paperwork, and the real problems, like a wrong policy answer or a customer left without a next step, hide inside the average.
This post shows how to build a scorecard small enough to use and sharp enough to find problems: what to put on it, how to weight it, how to keep reviewers consistent, and how to turn scores into coaching rather than ranking.

Start from outcomes, not from a template
A scorecard should reflect what a good contact means for your customers and your business. Before choosing criteria, write down a few sentences describing it. For example: the customer got a correct answer, knew what happens next, and was treated with respect, and the agent followed the rules that protect the company.
From that, pick four to six categories. A common, useful set:
- Accuracy. Was the information correct and complete for this case?
- Resolution. Did the contact move the issue forward, with a clear next step and owner?
- Policy and process. Did the agent follow the required steps, such as identity checks, escalation rules and what they are and are not allowed to promise?
- Communication. Was it clear, easy to follow and in the right tone for the channel?
- Handling. Was the contact tagged, documented and handed off in a way the next person can use?
More than six categories turns reviewing into box-ticking. If a criterion would never change a coaching conversation, drop it.
Separate the things that can fail a contact
Not all criteria weigh the same. A friendly reply with the wrong answer is worse than a curt one with the right answer. Handle that with two tiers:
- Critical items. Wrong information, breaking a policy, missing a required step, rude behavior. Any failure caps the contact at a fail, whatever the rest looks like.
- Scored items. Everything else, weighted. Accuracy and resolution get most of the points; tone and formatting get fewer.
Write the critical list explicitly and keep it short. When reviewers have to guess what counts as critical, the scores drift apart.
Weights should come from judgment about impact, and you will adjust them. Start simple, such as accuracy worth the most, resolution next, and policy and communication sharing the rest, then look at whether the scores separate good contacts from bad ones. If everything lands within a few points of each other, the scorecard is not discriminating, and the fix is sharper definitions, not more items.
Write the criteria so two people agree
A criterion like "showed empathy" will get a different score from every reviewer. Rewrite each one as something visible in the text or the call:
- Weak: "Used a professional tone."
- Better: "Addressed the customer's stated problem in the first two sentences and did not use blame language."
For each item, add one example of a pass and one of a fail, taken from your real contacts with details removed. These examples are the most valuable part of the scorecard, because they settle arguments before they start.
Allow "not applicable." A password question has no escalation step, and forcing a score there inflates the average.
Sample deliberately
Reviewing random contacts is a start, and it hides rare, costly problems. Combine:
- a random sample per agent, so each person gets a fair read
- a targeted sample from risky categories, such as account access, policy exceptions and complaints
- contacts flagged by customers or by the agent as difficult
Keep the numbers realistic. A lead who reviews a few contacts per agent per week and discusses them is worth more than a team that reviews many and discusses none. Decide the volume you can sustain, write it down, and keep to it.
Calibrate every month
Even good criteria drift. Once a month, have everyone who scores, including leads and any QA analyst, score the same five contacts on their own. Then compare line by line. Where two people differ by more than a level, the criterion needs a sharper definition or a new example.
Do this before you use scores in performance reviews. Scores that depend on which reviewer drew the contact are noise, and agents know it. If you cannot get reviewers to agree within a reasonable margin, use the scorecard for coaching only until you can.
Use the scores to coach, not to rank
The point of the scorecard is the conversation after it. A short format works:
- Pick one contact the agent did well and say why, using the criteria.
- Pick one where a criterion failed, show the exact line, and ask how they would handle it next time.
- Agree one change to try before the next review.
One change at a time. Handing someone six improvements guarantees none.
Look at patterns across agents too. If half the team fails the same item, the problem is the knowledge base, the macro or the training, not the people. This is where a scorecard pays back: it finds the shared cause. A canned reply that keeps scoring low on communication, for instance, is a case for rewriting the template.
Do not publish a leaderboard of scores. It rewards gaming the sample and punishes agents who take the hardest contacts.
Use the same rubric for AI and for people
If an AI assistant answers some of your contacts, score those with the same categories, with extra attention to accuracy and to whether it handed off at the right moment. A shared rubric lets you compare and avoid a separate, looser standard for automation. Reviewing AI answers before customers see them covers how to run that review.
Where RyteDesk fits
If you want to hold an AI assistant to the same standard as your team, you need to read what it says and check it before customers do. RyteDesk is in early access, built for mid-market support teams. It answers on chat, email and phone from the knowledge and policies you provide, tells callers it is an AI, and hands off to your team or takes a message when it cannot help. You can read every conversation and test it in a preview that sends nothing to customers. Join the waitlist to try it; it is free to join.