A demo is a controlled environment. The agent knows what is coming. The prospect cooperates. The audio is clean. Then production arrives with speakerphone echo, people who interrupt mid-sentence, voicemail trees, and a calendar connection that fails without telling anyone.
That distance between staged performance and live performance is where pilots die. IDC research shows that 88% of AI agent pilots never reach production. The models are capable. The evaluation process is what breaks.
If your acceptance test is a quiet-room demo, you tested a script. You did not test a system.
Why the Demo Numbers Collapse
Audio conditions alone explain a large share of the drop. One production analysis found the same speech recognition API hit 92% accuracy on clean headsets, 78% in conference rooms, and 65% on mobile calls with background noise.
Transcription errors compound. In a multi-turn conversation, one misread turn corrupts every turn after it. A small slip in minute one becomes a wrong booking in minute four.
Averages make this worse. A 90% average success rate can conceal a specific cohort failing at 40%. One accent group or one noisy channel carries most of the failures while the headline number looks fine. That cohort is small in your pilot and large in production.
So the question for your pilot is simple: which specific calls must the agent survive, and what evidence proves it did.
The Eight Scenarios to Test Before You Call It a Win
Each scenario below is a requirement to test. For each one, write the expected behavior before the call and define what evidence confirms it.
1. The person answers and gives a short, blunt reply
"Yeah, who's this?" The recording should show the approved opening and a relevant response, with no scripted monologue plowing past the brush-off.
2. The person interrupts mid-sentence
The agent stops or yields according to the configured turn-taking behavior. Review what the person actually heard, which means listening to the audio.
3. Speakerphone, noise, or overlapping speech
This is where recognition accuracy drops hardest. Verify the conversation stayed usable and the captured data stayed accurate under degraded audio.
4. The person pauses or restarts an answer
The agent waits. It does not invent an answer to fill the silence, and it does not lose the task.
5. Voicemail or a phone menu
The configured branch fires. Any keypad action gets verified through the action records, since the agent reporting success is a claim, and the record is the evidence.
6. The person asks for a human
The transfer, callback, or escalation path shows up in the final record. A promised handoff that never lands is a failed call wearing a polite transcript.
7. The calendar or connected action fails
This is the silent killer. The result must stay flagged as unresolved or follow the recovery path. A false "booked" status is the single most expensive failure mode in an outbound pilot.
8. The person declines or asks for no further contact
The final state must match your contact-restriction rules exactly. This scenario is a compliance requirement, and it belongs in every test matrix.
Why Transcripts Alone Are Not Evidence
A transcript is the system's belief about what happened. Audibility, talk-over, clipped speech, and unnatural pauses never appear in text. The person on the other end experienced all of them.
Review authorized recordings alongside the transcript. Then go one layer deeper and check the connected record: the attendee, the time, the booking status, the follow-up state. A call is complete when the claimed outcome and the system of record agree.
Execution that leaves no verifiable trace is theater, and you have no way to tell the difference until a customer does.
Make the Checks Repeatable
Run the matrix per scenario and per cohort, since averages hide the segment that will generate your support tickets. Keep a baseline with the scenario, the configuration version, and the accepted evidence.
Then rerun affected scenarios after every prompt, model, voice, or integration change. A configuration change resets your confidence to zero until the evidence restores it.
Agree on rollout blockers before the pilot starts: wrong caller identity, ignored contact restrictions, invented availability, and completions reported without evidence. Name the person with authority to stop the campaign.
A pilot passes when all eight scenarios hold under real conditions, with recordings and records that agree. Anything less is a demo with a longer runtime.