Audit Discovery Calls Like RevOps: Turn Playbook Criteria Into Deterministic Scoring and Coaching Gaps
Discovery call quality is rarely the real problem; the real problem is evaluation, because teams can’t measure buyer understanding against their playbook, so coaching runs off noise. The rep got positive feedback because the call felt energetic, while the deal stalled because the buyer never articulated the consequence of their problem, and nobody ever connected those two facts.
Auditing discovery calls like RevOps means fixing the measurement layer first. That means turning playbook criteria into structured evaluation Bricks, scoring every call against them, and routing the results into coaching workflows where the evidence is already attached. Reviewing a sample of calls and coaching from impressions produces coaching that is inconsistent, hard to scale, and disconnected from the deal outcomes it was supposed to improve.
The evaluation problem, not the quality problem
Discovery call quality isn’t a coaching volume problem, because most teams already review calls; the problem is what gets measured during the review. Subjective impressions of rep delivery, notes on whether the call felt like a good conversation, and summaries of what was discussed aren’t the same as a consistent assessment of whether the buyer understood the qualification logic your playbook requires.
The test of a good discovery evaluation is whether two separate reviewers assessing the same transcript reach the same conclusion. On most teams they don’t, because the criteria are implicit and the evidence isn’t tracked. When the criteria are implicit, coaching reflects whoever did the review that week, not a stable standard. And when coaching is inconsistent, the only thing that changes is the volume of feedback, not the outcomes.
Why freeform AI scorecards repeat the same mistake
Freeform AI scorecards built on a general prompt, "assess whether this was a good discovery call," inherit the same problem as subjective human review. The criteria are still implicit. The model interprets what "good" means and returns a score, but the interpretation varies across runs, across model versions, and across calls the prompt author never anticipated. The score looks authoritative because it came from an AI, but it reflects the same unconstrained judgment that subjective review does.
The evaluation contract post covers this in detail. AI scorecards disagree not because the model is unreliable but because the prompt never specified what evidence would confirm the criterion. A scorecard built on a vague prompt produces vague outputs regardless of the model running it, and coaching built on those outputs produces inconsistent results regardless of how frequently it happens.
The reframe: buyer understanding, not rep behaviour
The thing worth improving in discovery is buyer understanding: whether the buyer grasped the qualification and next-step logic your playbook requires, and left evidence of it in the transcript. Rep delivery only matters insofar as it produces that evidence. A rep who delivered a polished, energetic call without getting the buyer to articulate a business consequence didn’t run a good discovery call. A rep who spent the first fifteen minutes on background and then got the buyer to name a timeline, a consequence, and a decision-maker ran a better one. The transcript evidence is different, and the score should be different.
This reframe changes what you are evaluating: rather than assessing whether the rep asked the right questions, you are assessing whether the buyer produced the right evidence. The distinction matters because it is the buyer side of the transcript that determines whether a deal has genuine qualification, not the rep side. Buyer-side evidence is measurable, consistently, from the transcript, without inference about the rep’s intent or effort.
Build evaluation Bricks from your playbook
Turning a playbook into a scoring system starts with identifying the observable, evidence-grounded criteria that confirm each qualification element. For each criterion, the design question is: what would a buyer explicitly say in the transcript that confirms this element is present, and what would a reliable null look like when it is absent? A criterion that has a clear answer to both questions can become a Brick. Criteria that require the evaluator to interpret intent or infer quality can’t be scored consistently and should be redesigned before they are added to the schema.
Concrete examples of criteria that work well as Bricks include: whether the buyer named a specific business consequence of their current problem, whether they identified a timeline for resolution, whether the economic buyer was named on the call, and whether a specific next step was committed to with a date. Each has clear transcript evidence when present and a reliable null when absent, and a typed output, boolean or extracted string, that can be written to a CRM field without transformation. More interpretive criteria, such as whether the rep demonstrated value or whether the buyer seemed engaged, aren’t discoverable from transcript evidence in a way that two reviewers would agree on, so they produce unreliable Bricks and should be excluded from the scoring schema.
Once the Bricks are defined, they are grouped into a Kit that represents the discovery evaluation schema. Every call scored against the same Kit version produces an output with the same fields, the same types, and the same possible values, which is what makes the results comparable across reps, managers, and time periods.
Score every call and route gaps into coaching
With the discovery Kit in place, every call produces a structured output: a JSON object with one scored field per Brick. That output can be written to CRM opportunity fields, pushed to a reporting table, or routed to a coaching workflow. The workflow logic can be simple: if a deal has had two discovery calls and the economic buyer field remains null on both, that is a gap. If a rep’s discovery calls have a consequence field fill rate below the team average, that is a coaching signal. Neither observation requires a manager to review calls manually, because the evidence is already extracted and stored.
Rep-level gaps and team-level gaps serve different purposes: a rep-level gap, a specific criterion consistently absent across one rep’s calls, drives a one-to-one coaching conversation with specific transcript evidence attached, whereas a team-level gap, a criterion missing on a high percentage of calls across the whole team, drives a change to the playbook, to training, or to the enablement materials that introduce the criterion. The behavioural change post covers how structured call outputs connect to the kind of coaching that actually changes what reps do next week.
Semarize turns discovery call transcripts into typed, evidence-grounded scores against the playbook criteria you define, ready for CRM enrichment, coaching workflows, and rep and team gap analysis.
Common questions
How do we define buyer understanding in a way that can be scored consistently?
The test is whether the criterion is answerable from explicit buyer statements in the transcript, not from inference about their mindset or engagement level. A buyer who says "our current tool costs us about four hours a week in manual reporting" has produced evidence of a consequence. A buyer who "seemed interested" hasn’t produced anything scoreable. Buyer understanding criteria are defined by what the buyer would explicitly say when they genuinely have the understanding, and what would reliably be absent from the transcript when they don’t.
What if our playbook criteria are vague or written as best practices?
Most playbook criteria are written as rep behaviour instructions, not buyer outcome criteria. "Ask open-ended questions" and "build rapport early" are instructions to the rep, not evidence thresholds for the buyer. Converting them to Bricks requires asking: what would the buyer say if this instruction were followed well? If the rep asked open-ended questions about the buyer’s problem, the buyer would describe the problem in their own words. That description is the observable evidence. The Brick criterion is whether the buyer described the problem in specific terms, not whether the rep asked open-ended questions.
Do we need to replace all call reviews, or just change what we measure?
Structured scoring doesn’t replace manager call review for complex coaching situations. It changes what triggers a review and what the reviewer is looking for when they listen. Rather than reviewing a sample of calls to form an impression, managers can use the structured output to identify which specific calls and criteria need closer attention. The review becomes targeted rather than exploratory, and the feedback it produces is grounded in specific transcript evidence rather than an overall impression of the call.
Where should the structured scores live so RevOps can use them in workflows?
The most useful location for discovery scores is on the CRM opportunity record, as typed fields that map to the Brick output types: boolean fields for presence criteria, number fields for quality scores, and text fields for extracted strings. When the scores live in the CRM, RevOps can use them in pipeline views, forecasting models, and automated workflow triggers without a secondary data export step. Scores that require opening the conversation intelligence platform to inspect don’t drive workflows.
Continue reading
Read more from Semarize
AI scorecards are theatre unless they measure customer understanding
Most AI call scorecards measure what the rep did - agenda set, questions asked, next step mentioned. That's measuring inputs. What actually matters is whether the buyer understood anything. The two are not the same thing, and the gap between them is where scorecard theatre lives.
100% QA Scoring Without Manual Review: Deterministic Rubrics for Every Call
Manual QA sampling at 2–5% has two problems: coverage and consistency. Automated scoring with deterministic rubrics solves both - every call gets scored the same way, with no reviewer required to generate the result. The shift isn't just efficiency - it changes what coaching is built from and turns compliance verification from sampling into complete coverage.
MEDDICC Scoring From Discovery Calls
Most MEDDICC data is stale before it reaches CRM. Reps update fields from memory after the call, introducing timing gaps and sampling bias that make qualification scores unreliable. Extracting MEDDICC signals directly from transcripts fixes the data freshness problem that better training never will.