Bricks and Kits: How to Build a Stable Evaluation Schema That Never Drifts

The most common failure in AI call evaluation isn't a bad model - It's that the evaluation criteria were never properly defined, so they shift silently with every model update and prompt change, until scores from three months ago no longer compare to today's and the same transcript run twice returns different results. Teams add criteria informally, drop them quietly, and downstream systems fill up with gaps nobody can explain.

Bricks and Kits are how Semarize fixes this. Bricks are single evaluations: one question, one typed answer, the same every time. Kits are groups of Bricks that together define how you evaluate a transcript. Lock the Kit and the schema stops moving: every transcript is scored against the same criteria, in the same way, until you explicitly change it.

What a Brick actually is

Think of a Brick as a question you ask about every call, with a fixed answer format. "Did the buyer mention a budget constraint?" returns yes or no. "How well did the rep run discovery?" returns a score from one to five, against defined anchors for each level. "Did a competitor come up?" returns the name as a string, or null if none was mentioned.

The answer format is part of the Brick's definition, not something the model decides at runtime. That's what makes results consistent: the same transcript scored twice against the same Brick returns the same answer. The same call reviewed by two different people, working from the same Brick, reaches the same conclusion. If that repeatability breaks down, the Brick needs to be more specific.

Bricks are reusable across Kits: a discovery quality Brick defined once can appear in a sales coaching Kit, a MEDDIC scoring Kit, and a pipeline health Kit, applying the same criteria and returning the same output in each. When you update the Brick, the change carries through every Kit that includes it, but only within a new Kit version. The current version keeps using the Brick as it was when locked.

What Kits do for schema stability

Kits are full evaluation schemas, one per use case: a discovery Kit covers everything you need to know whether a discovery call qualified the opportunity, a QA Kit covers the compliance and process requirements a call needs to meet, and a win/loss Kit captures the signals that explain why deals go the way they do.

Running a Kit against a transcript returns a JSON object with one field per Brick. Every call processed against the same Kit version comes back with the same shape: the same fields, the same types, the same possible values. That consistency is what makes the output usable downstream, whether it feeds CRM enrichment, warehouse tables, or reporting dashboards, without anyone reshaping it first.

Versioning is what keeps scores comparable over time. When a Kit version is locked and deployed, the criteria are fixed. Two months later, calls run against the same version are scored against identical criteria. When the evaluation logic needs to change, you create a new Kit version and the old one keeps running until you retire it. Historical calls can be rescored against the new version if you need comparable data across both.

The four ways schema drift happens, and what it costs

Most teams don't choose to let their evaluation schema drift; it happens in four common ways. Model updates: the AI model gets updated and its behaviour shifts, changing outputs without any change to what you asked it to evaluate. Prompt edits: someone tweaks the instructions and outputs shift in ways that are hard to predict and impossible to detect without running the same calls before and after.

Criterion creep: new evaluation criteria get added informally over time, old ones get quietly dropped, and the schema your downstream systems expect starts diverging from what the evaluation actually produces. Field renames: a criterion gets restructured, the CRM field mapped to the old name stops receiving values, and the gap only surfaces months later when a report starts showing unexpected nulls.

The cost of all four is a dataset that looks continuous but isn't. Time-series analyses comparing this month's scores to last month's are comparing values produced against different criteria, which makes the trend meaningless. Before-and-after coaching assessments show noise instead of signal. Pipeline health models produce outliers nobody can explain, and the usual response is to distrust the data rather than fix the schema.

Designing Bricks that don't drift

The most common reason a Brick drifts is that the criterion relies on model interpretation rather than evidence in the transcript. "Did the rep demonstrate strong rapport?" requires the model to decide what strong rapport looks like, and that decision shifts as the model is updated. "Did the buyer mention returning to this problem at least once unprompted?" has an observable answer in the transcript.

The test is repeatability: run the same transcript against the Brick twice, with a gap between runs. Same answer both times means the Brick is stable; different answers mean the criterion is too interpretive and needs to be more specific. For teams building their first Bricks, starting with the most concrete, evidence-grounded criteria and working toward more interpretive ones as the programme matures is more reliable than trying to capture nuanced qualitative judgements from the start.

Starting from what you already score manually

If your team already reviews calls against any kind of checklist or rubric, each item on it is a candidate Brick. The exercise is converting each item from a description into a typed criterion with a fixed answer format. "Rep built good rapport" becomes a yes or no answer to "did the buyer mention returning to this problem at least once unprompted". The items that can't be converted without ambiguity are the ones that will produce drift, and they need to be tightened before becoming Bricks.

Once the Bricks are defined, grouping them into a Kit is straightforward: put the criteria that belong to the same use case into the same Kit. A discovery Kit, a QA Kit, and a MEDDIC Kit are three separate schemas because they serve different purposes and different teams. Mixing them into one large Kit makes the output harder to route and the schema harder to maintain. Start with five to eight Bricks per Kit, and add only when a missing criterion is demonstrably affecting the use case.

Semarize uses Bricks and Kits to keep call evaluation schemas typed, versioned, and stable enough for CRM enrichment, coaching workflows, and production reporting.

Start building →

Common questions