A sample order and timer at an operations workbench

procurement · governance

How to write acceptance criteria before buying an AI pilot

Give AI pilot bidders the same acceptance cases. Compare the work your team retains and the evidence a supervisor needs before approving a wider rollout.

Remolda Team6 min read

A supplier can give a convincing demonstration of an AI workflow and still leave an operations manager unsure whether to buy it. The missing piece is often an acceptance test written in the language of the desk that will use it. Before asking bidders to price a pilot, describe one piece of work your team would accept on a busy day and one it would send back. Those examples give the pilot a finish line both sides can recognise.

Buy a piece of work your supervisor can inspect

Consider a parts distributor that receives quote requests by email. Buyers send catalogue numbers when they have them. Sometimes they send a description from the label on an old machine. The sales coordinator looks up the part and prepares a draft reply for a colleague to review. A project brief that asks for “AI email automation” could describe almost anything from sorting messages to issuing binding prices.

Write down exactly where the pilot stops. For this example, the buyer might want a draft that names the requested part and shows what information still needs confirmation. A salesperson remains responsible for the price and for sending the reply. Give that description to the salesperson and ask them to mark any work they would still have to do. If they need to look up the very same part again, the draft has failed an important practical test.

Avoid building the first acceptance case around the easiest message in the inbox. A perfectly formatted request proves little about the messages that occupy a supervisor's afternoon. Pick an ordinary source of rework, such as two catalogue entries that share a description. Replace customer identifiers with invented details and preserve the ambiguity. That makes the sample safe to share while keeping the commercial question intact.

Agree on what a good stop looks like

A careful pilot should get credit for pausing when it lacks a fact needed to produce the next output. In the parts example, a request could refer to either a left-hand or a right-hand fitting. It would be a mistake for the buyer to pay a supplier for maximising the number of completed drafts while measuring nothing about those ambiguous cases. Specify which question the salesperson needs answered before a draft can proceed.

Work out who sees the paused item. A reply that says “needs review” is useful only when a person responsible for the queue can find it. Ask the vendor to demonstrate the case from the reviewer's seat. The reviewer should see the original wording and the detail that remains uncertain. If they have to hunt for the message in a separate shared mailbox, include that time in the pilot review.

  1. Select an ordinary request with a clear expected output. Have a supervisor write the expected result before the vendor begins.
  2. Select an ambiguous request that should reach a reviewer. Record the information the reviewer needs to continue.
  3. Change one approved reference item during the pilot. Check whether the team can identify drafts made under the earlier version.
  4. Let a colleague who missed the demonstration take over an open item and explain the decision they would make.

For a written assessment of a candidate workflow, send Remolda a pilot brief. Include a fictional request and name the role that approves the final result. The AI readiness review is a starting point when the team has yet to establish which systems and staff a pilot would need.

Put staff time in the measurement

Pick up a stopwatch only after the supervisor agrees where work begins and ends. Time spent drafting a reply and time spent waiting for a customer answer are different. Record them separately. Say a request arrives on Monday with the wrong catalogue number. The customer supplies a usable number on Thursday. A timer left running between the two messages records three days that no amount of faster drafting could have saved.

Give setup its own line on the buyer's timesheet. Someone has to identify the current catalogue and explain how the team handles discontinued parts. That work might also be valuable outside the pilot. It still consumes staff time and the manager should be able to see it. Ask the supplier to record assumptions about access to staff in the quote so the pilot is priced against the same working conditions across bidders.

The person approving the purchase may want a simple cost comparison. On the worksheet, put the number of messages that reached a supervisor beside the time saved. Separate the clean catalogue numbers from the requests somebody had to read twice. The manager can then decide whether the evidence covers the work the business actually wants to buy. Extrapolating a quiet Tuesday across a whole year would hide the awkward part of the job. If the business case depends on handling the winter rush, include a representative busy-period sample before making that commitment.

Ask for handover material while the vendor is still bidding

A pilot leaves behind more than the demonstration. Ask for the approved example cases in a form the buyer can read later and for the instructions the team used when deciding whether to pause work. Record who can update a reference item. If the supplier keeps those materials in its own workspace, agree how the buyer can obtain a usable copy when the engagement ends.

Schedule one handover case for a colleague who normally covers the sales desk during holidays. Open a draft with an unfamiliar part number and leave the supplier out of the room for a few minutes. The colleague needs to find the catalogue entry and say whether they have authority to release the price. When they get stuck, ask them to show you the screen. A missing link in the instructions will be easier to see there than in a presentation.

When comparing bids, Remolda's AI vendor selection service describes the procurement side of the work. Keep the acceptance cases alongside that comparison. A lower quote that assumes the buyer will prepare all the cases can make sense if the team has already done so. A higher quote may include a supervisor workshop that the business would otherwise have to organise internally.

Close the pilot with a decision somebody owns

Set up a short written decision record before the first case enters the pilot. At the end, the operational owner can approve a wider rollout, request one more test or stop the project. Ask the owner to attach the case that drove the decision. If the part number was right and the colleague could not find the sign-off instructions, the supplier has a specific handover issue to fix. If the draft named the wrong fitting, look at the reference the tool used before agreeing to another trial.

Leave an unresolved case open on the review sheet if one still matters. Write down what it would take to settle it. The next manager can then decide whether an extra day with a supervisor is worth buying. Request a written pilot scope with the workflow your team wants to test and the person who will decide whether the result is ready for use.

Share

Related insights