Walking into an AI voice demo without a scorecard is like hiring someone without a job description. The vendor will impress you. You will leave with no way to compare what you saw to the next vendor, because you never defined what passing looks like.
The numbers support the caution. According to Forrester, 67% of AI buyers said the product performed worse than the demo within 90 days. A demo is a controlled performance. Your deployment will run under conditions the vendor did not script.
The scorecard fixes this. It has four sections, and you write all of them before the first call with any vendor.
Section 1: The Calling Job
Write the job in one sentence. Who gets contacted, why the contact is permitted, what the agent should accomplish, and what a person on your team does next.
This sentence does the work a job description does in hiring. It turns "that sounded impressive" into "that matched the job" or "that did not."
A quote for an inbound receptionist tells you nothing about an outbound qualification campaign. When you hand the vendor your calling job, you force the demo onto your terms instead of theirs. Every scenario they show you either maps to the job or it does not count.
Section 2: Evidence Paths, Written Beside Each Feature
Features describe what the system can do. Evidence paths show you what actually happened on a call. You need the second one to run the first one responsibly.
For each outcome the vendor claims, ask to see the trail:
- The recording and transcript for the conversation itself
- The connected record for any booking, with attendee, time, and status
- The event history when an action fails, so nothing reports as complete without proof
- The final state when a person declines or asks for a human
Execution that leaves no trace cannot be reviewed, corrected, or trusted at scale. If the vendor cannot show you where the evidence lives for your specific workflow, mark that row unresolved and move on. Unresolved is a valid score. Guessing is the failure mode.
Section 3: Budget Lines Confirmed in Writing
The headline per-minute rate is the start of the price, rarely the whole price. Gartner found the average AI platform carries 3 to 5 hidden cost layers, including API fees, storage, and per-seat pricing. For voice specifically, a complete cost includes speech-to-text, text-to-speech, model usage, telephony, storage, and compliance, and vendors bundle these pieces differently.
Your scorecard needs a budget line for each of these:
- Platform fee, billing period, and minimum commitment
- Billable minutes, rounding rules, and voicemail or transfer charges
- Lead data, enrichment, and replacement of unusable records
- Implementation, integration work, and acceptance testing
- Ongoing review time and exception handling
Mark each line as included, separately billed, not required, or unresolved. Accept only written amounts. A verbal "that's included" has a short memory once the contract is signed.
Section 4: Rollout Blockers the Vendor Must Prevent
Before the pilot, agree on the conditions that stop everything. Suggested blockers: wrong caller identity, ignored contact restrictions, invented availability, repeated speech loops, and critical actions reported as complete without evidence.
Then ask the vendor to show you how their system prevents each one. Hard constraints matter more than dashboards here. Spending caps that hold, quotas that reserve capacity, and systems that fail closed when something breaks.
Assign one person on your team the authority to stop activity when a blocker appears. A blocker list without an owner is a suggestion.
The Decision Lives Outside the Demo
The abandonment data makes the stakes concrete. S&P Global found AI initiative abandonment before production jumped from 17% in 2024 to 42% in 2025. Buyers who choose well apply a structured scorecard identically across every vendor, covering pricing clarity, integration surface, call quality under load, and observability.
Score every vendor against the same four sections. Treat the demo as one input, then require a written quote, a pilot against your real calling job, and evidence you can inspect for every claimed outcome.
The vendor that wins your demo earns a pilot. The vendor that passes your scorecard earns the contract.