Two review queues.
Only one of them measures anything.
Every vendor says a human reviews a portion of the output. Almost none say how that portion was chosen, and the choice is what decides whether you get a number or a promise.
9 September 20265 min read
The reasonable thing to do is the wrong thing to measure with
You have a system scoring calls against a rubric, and a reviewer with time for a fraction of them. The obvious move is to send that reviewer the calls the system is least sure about: the muddy transcripts, the variables it could not determine, the calls where speaker separation looked shaky.
That is the right thing to do, and you should do it. It concentrates the reviewer's time where errors actually live, so the file you hand over is measurably better than the raw output.
What it cannot do is tell you how accurate the system is. You deliberately selected the hardest cases. The error rate you observe there is the error rate of the hard cases, not of the work. Report it as accuracy and you will understate your own system — and if you correct for that by guessing, you are no longer reporting a measurement.
So there are two queues, and they do different jobs
Random sample
To measure
Drawn without looking at confidence, difficulty or anything else. If the reviewer picks which calls to check, or the system picks for them, the estimate is biased and stops being a number you can defend.
Confidence-driven queue
To correct
Everything the system flagged: low-confidence transcription, non-determinable variables, poor diarisation. It improves the deliverable and it is useless for computing accuracy.
Run both, report them separately, and never average them together. Mixing the two is not a rounding error — it destroys the only figure in the process that a buyer can check.
How big the measuring sample has to be
Smaller than people expect, and — the part that surprises everyone — it does not grow with your volume.
About 400 randomly drawn calls give a margin of roughly ±3% at 95% confidence. That is the same 400 whether you process 5,000 recordings a month or 40,000, because the precision of a proportion depends on the size of the sample, not on the size of the population it came from.
The practical consequence is that a measuring sample should be specified as an absolute number, not as a percentage of volume. A fixed 2% sounds rigorous and is simply more expensive at scale for no additional certainty — you are buying the same margin twice.
What this changes in a proposal
"We review a sample" is a description of a process. It commits the vendor to activity, not to a result, and it is compatible with any level of quality.
"Accuracy measured at X% on a random sample of 400 calls, ±3% at 95% confidence" is a claim about the deliverable. It can be reproduced, it can be wrong, and it can be checked against the next batch.
The second one is harder to write, which is exactly why it is worth asking for. A vendor who cannot tell you how their sample was drawn has not measured anything — they have reviewed some calls.
The side effect nobody plans for
The reviewer's corrections accumulate, and after the second month they stop being corrections and start being evaluation data.
You can see which variables fail, and under what conditions: short calls, a particular queue, a specific accent, audio from one recording platform. Calibration stops being an argument about what the model probably does and becomes a list of what it demonstrably did.