Why Coaching From AI Scorecards Usually Fails (And What the Score Actually Needs to Measure)

AI scorecards mostly measure whether reps completed the right activities on a call: did they ask about timeline, did they mention pricing, did they confirm next steps. These are rep-side inputs, and they’re the wrong unit. A rep who asked about timeline and got a vague answer hasn’t qualified the deal. A rep who didn’t ask about timeline but whose buyer volunteered a deadline has. The score that only measures the rep can’t tell those two calls apart.

Coaching from activity-based scores produces coaching that tells reps what to do more often, not what to do differently based on what the buyer actually said. That is why conversation intelligence programmes that are nominally working, running every call through a scorecard and producing weekly coaching reports, don’t produce measurable improvement in deal outcomes. The scorecard is measuring the wrong side of the conversation.

What AI scorecards usually measure

The default architecture of an AI scorecard is a prompt that asks the model to assess the call and return a score or a set of ratings. The criteria are typically derived from a sales methodology: MEDDIC, SPIN, BANT, or a proprietary playbook. The model reads the transcript, interprets whether the criteria were addressed, and returns a score. This works well for high-level summaries, but not as a foundation for coaching, because the model is being asked to make a judgment rather than find evidence.

When the criterion is "did the rep ask about budget," the model can usually answer that from the transcript reliably. When the criterion is "did the rep demonstrate value," the model is interpreting what demonstrating value means and returning a judgment that will vary across runs, across model versions, and across calls the prompt author didn’t anticipate. That variance means the scores from one week can’t be compared to the scores from the previous week without knowing whether the model or the prompt changed in between. Coaching built on scores that can’t be compared over time has no way to confirm that anything actually improved.

The problem with measuring rep activity

Activity-based scoring measures the rep’s behaviour in isolation from the buyer’s response. A rep who asked "what would happen if you didn’t solve this problem?" gets credit for asking the consequence question. A rep whose buyer spontaneously described a consequence without being asked the question directly gets less credit, or none. But the buyer who described the consequence is more qualified than the buyer who answered a rote question with a polite non-answer.

The unit of qualification is what the buyer said, not what the rep asked. Deals where the buyer has named a consequence, identified a timeline, and confirmed the economic buyer’s involvement are qualified regardless of which specific questions the rep used to get there. Deals where the rep followed the playbook script and got no substance from the buyer aren’t qualified, regardless of the activity score. Coaching off activity scores improves script adherence, whereas coaching off buyer-side signals improves deal quality.

What the score actually needs to measure

A scorecard that changes coaching behaviour needs to evaluate buyer-side evidence: what the buyer said, what they confirmed, and what they didn’t address. Each criterion in the scoring schema should have a clear answer to the question: what would a buyer explicitly say in the transcript that confirms this element is present? That question anchors the criterion in observable evidence rather than interpretation, which is what makes the score consistent across runs and useful for comparing calls over time.

Practical examples of buyer-side criteria include: whether the buyer described a specific business consequence with a named impact, whether they identified a decision timeline in their own words, whether they named the economic buyer by role or name, and whether they committed to a specific next step rather than a vague follow-up. Each of these is recoverable from the buyer’s turns in the transcript without requiring the evaluator to interpret rep intent. Each produces a typed output, boolean or extracted string, that can be stored in a CRM field and queried across the pipeline.

This is what Bricks in Semarize are designed to produce: a single evaluation criterion with a defined question, a defined output type, and explicit scoring logic anchored in transcript evidence. Applied consistently across calls, they produce scores that are comparable over time because the criterion hasn’t changed, and usable for coaching because the evidence behind each score is recoverable from the transcript.

Why consistency is a prerequisite for coaching

Coaching requires a baseline: to know whether a rep improved on discovery quality after a coaching conversation, you need a score from before and a score from after that were produced by the same criteria. If the scorecard drifted between the two measurement points, even slightly, because the model was updated or the prompt was adjusted, the comparison is unreliable and the coaching effect can’t be confirmed.

Most teams aren’t aware that their scorecards drift, because drift is silent. The scores keep coming, the dashboards keep updating, and nobody notices that the criteria changed until the trend line produces anomalies that nobody can explain. The evaluation contract post covers the four common ways scorecard schemas drift and why versioning the evaluation logic is the only way to prevent it. For coaching, the implication is concrete: a coaching programme built on drifting scores has no reliable signal to measure progress against.

What coaching from buyer-side signals looks like

When the scorecard measures buyer understanding rather than rep activity, the coaching conversation changes. Rather than telling a rep to ask the consequence question more often, the coaching conversation starts with: on your last five discovery calls, the buyer described a consequence on two of them. Here are the two transcripts where they did. Here are the three where they did not. What was different about the calls where the buyer went there on their own?

That conversation is about outcomes, grounded in evidence the rep can verify themselves. It doesn’t require the manager to have listened to all five calls. It requires a score that is trustworthy enough to be the starting point of a serious conversation, and evidence specific enough that the rep can look at the transcript and understand what the score reflects. The behavioural change post covers the conditions that distinguish coaching that changes what reps do from coaching that produces acknowledgement and then no change.

The score only works as a coaching tool if it measures something real, produces the same result when applied twice to the same call, and reflects the buyer’s understanding rather than the rep’s script adherence. Most AI scorecards don’t meet those conditions. The ones that do are built on explicit, evidence-grounded criteria rather than freeform prompt evaluations, and they are versioned so the criteria can’t change without a deliberate decision to change them.

Semarize turns scorecards into evidence-grounded Bricks so coaching starts from buyer-side signals, stable criteria, and transcript evidence reps can verify.

Start building →

Common questions