When an AI voice pilot ends and the room disagrees on whether it succeeded, the data rarely caused the disagreement. The definitions did. Someone counted booked meetings. Someone else counted meetings that happened. A third person counted opportunities sales actually accepted. Three honest numbers, three different stories, zero agreement.
Those numbers measure different events. If you never wrote down which one the pilot had to move, the pilot was a demo with a budget.
The research backs this up. RAND found that 84% of AI project failures trace back to leadership decisions rather than technical limits. Teams deploy first and define success later, and later never arrives.
Start With the Purchase Question
Before any configuration work, write one sentence: who is being contacted, why the contact is permitted, what the agent should accomplish, and what a person does next.
That sentence forces the decisions teams usually postpone. Warm follow-up and cold outreach belong in separate reports. A qualification campaign and a booking campaign have different acceptance evidence. One segment, one stable offer, one observation window.
Skip this step and the pilot answers a question nobody asked. MIT's Project NANDA found that 95% of generative AI pilots deliver zero measurable P&L return, and only 20% of pilots ever reach production. The organizations behind those numbers usually started with the tool instead of the question.
Separate the Four Numbers That Get Conflated
Four events live in every meeting-led campaign, and each needs its own status and its own evidence:
Booked meeting. A qualifying booking confirmed in the connected calendar record.
Held meeting. A booking with attendance evidence. People cancel. Calendars lie.
Qualified meeting. A held meeting where the attendee matches the buyer fit you agreed on in writing, judged by a named person.
Accepted opportunity. A deal sales takes under declared criteria, tracked on its own timeline.
Treating these as one number produces reports that look strong and mean nothing. A pilot can book thirty meetings, hold eighteen, qualify nine, and hand sales two opportunities. Every stage conversion in that chain is a decision input. The blended version hides all of them.
Write Acceptance Criteria Like a Requirements Document
A requirements document written before implementation defines what the agent is allowed to do, how success gets measured, and what evidence must exist. Mature frameworks assign launch targets, no-go triggers, and one reviewer who owns each metric.
Your version needs five parts:
1. The primary outcome. Pick it before you look at results. Qualified meetings held works well for meeting-led campaigns.
2. The acceptance owner. Name the person who decides whether a meeting counts. Committees produce averages. Owners produce decisions.
3. Honest denominators. Keep unique contacts separate from dialing attempts. A retry adds an attempt without adding a person. Label every rate with its denominator.
4. Quality blockers. Wrong caller identity, ignored contact restrictions, invented availability, actions reported complete without evidence. Any one of these stops the rollout regardless of the conversion numbers.
5. The cutoff date. Outcomes that have not matured get reported as pending, with the date stated. Unresolved stays unresolved. Guessing corrupts the whole record.
Agree on the Baseline and the Decision Rules
A pilot without a comparison produces a story instead of a result. Use comparable audiences, offers, calendar availability, and follow-up ownership. If you compare against historical performance, list what changed in the meantime, because those changes compete with the agent for credit.
Then write the three possible endings before launch. Expand when the observed workflow and economics meet your thresholds. Repair when the evidence points to a correctable issue. Stop when a critical blocker or missing evidence prevents an accepted result.
Set the thresholds from your own margins and capacity. Borrowed benchmarks measure someone else's task with someone else's denominator.
The Definition Is the Deliverable
Execution that leaves no trace is theater. The same holds for pilots. A pilot that ends in argument taught you nothing about the technology. It taught you that the team launched without agreeing on what the finish line looked like.
Write the purchase question. Split the four numbers. Name the acceptance owner. Set the decision rules. Do all of it before the first call goes out, and the end of the pilot becomes a reading of the record instead of a negotiation about it.
That document costs you an afternoon. Skipping it costs you the pilot.