AI agent pilot decision template: continue, change, or stop

The hardest part of an AI pilot is not the build. It is writing down the decision the evidence actually supports.

A demonstration can look promising while the operating proof remains incomplete. A metrics sheet can show better averages while hiding correction time, unowned exceptions, or a control failure. A meeting can end with general enthusiasm but no owner, boundary, review date, or stop condition.

This AI agent pilot decision template turns the evidence into one accountable record. Complete it for one workflow and one pilot period. If a fact is missing, write UNPROVEN. Do not silently widen the agent’s authority to make the result look more complete.

If you are earlier in the process, start with the AI agent readiness checklist, then use the 30-day AI agent pilot plan. Bring the completed AI agent pilot metrics scorecard into this decision review.

One-page AI agent pilot decision memo

Print or copy this worksheet into the operating record. Keep supporting evidence linked, but keep the decision itself on one page.

Pilot decision memo

1. Workflow / operating job: ___; business owner ___; agent-assisted step ___.
2. Pilot period and representative sample: From ___ to ___; ___ cases; routine and relevant exception types included ___.
3. Original baseline and target: Baseline ___; target ___; shared unit ___; source/evidence link ___.
4. Measured result in the same unit: Result ___; difference from baseline ___; material context change ___.
5. Acceptance without material correction: ___ of ___ reviewed outputs = ___%; agreed acceptance rule ___.
6. Human review and correction burden: Review ___ minutes/case; correction ___; exception handling ___; total human burden vs. baseline ___.
7. Exception rate and ownership: ___ of ___ cases = ___%; top exception ___; named owner ___; unresolved exceptions ___.
8. Control and prohibited-action result: Missed approvals ___; source-boundary failures ___; prohibited actions/attempts ___; disposition ___.
9. Evidence quality: Measured ___; observed but incomplete ___; assumed ___; stale ___; sample limitation ___.
10. UNPROVEN items: ___; why missing ___; consequence for this decision ___; evidence owner/date ___.
11. Decision: Continue / Change / Stop; evidence-based rationale ___.
12. Accountable decision owner and date: Owner ___; role ___; decision date ___.
13. Next authorized boundary: Allowed workflow/cases/actions ___; volume ___; human approvals retained ___; still prohibited ___.
14. Named changes: Source ___; instruction ___; scope ___; review step ___; exception path ___; no change ___.
15. Next review date and evidence window: Review ___; cases/time period ___; measures that must be available ___.
16. Rollback / stop condition: Trigger ___; who stops the pilot ___; rollback action ___; notification/evidence path ___.

The memo is complete only when the evidence, decision, owner, boundary, and next review can be understood without reconstructing the meeting from chat or memory.

Continue, Change, or Stop: a practical decision rubric

Do not use a universal pass percentage. The right threshold depends on the workflow, consequence, case mix, and agreed charter. Use the evidence against the pilot’s own predefined acceptance and control boundaries.

Continue

Choose Continue when the measured operating result supports the original objective, accepted quality and human burden are within the agreed range, representative exceptions reach named owners, required controls held, and the evidence is strong enough for the next bounded step.

Record exactly what continues. Name the case types, volume, source boundary, human approvals, next review date, and stop condition. Continue does not mean unrestricted rollout or removal of human authority.

Change

Choose Change when the operating job is still valuable but the evidence identifies a specific, testable design problem. Examples include a source gap, ambiguous instruction, scope that is too broad, recurring correction pattern, slow review step, or exception path that needs a clearer owner.

Name the change and start a new evidence window. Preserve the original result rather than blending it with the revised pilot. The changed version must earn its own decision.

Stop

Choose Stop when the business value does not justify the human burden, the evidence cannot support safe continuation, reliability is unsuitable for the consequence, exceptions remain materially unowned, or a predefined safety, approval, data, quality, or workload boundary is crossed.

Stopping is a valid operating decision. It prevents a compelling demonstration from becoming unmanaged production work.

A prohibited action or failed control can override favorable averages. Missing evidence remains UNPROVEN; it does not become positive evidence because the meeting needs an answer.

How to run the decision review in 30 minutes

Before the meeting: send the evidence pre-read

The owner should receive the completed pilot metrics scorecard, representative-case evidence, material corrections, exception log, control events, and known UNPROVEN items before the meeting. The meeting is for deciding, not discovering that evidence is missing.

Minutes 0-5: confirm the job and evidence boundary

Restate the workflow, pilot period, representative sample, original baseline, target, and authority boundary. Confirm that the result uses the same unit as the baseline. Label changed conditions and evidence gaps.

Minutes 5-15: review results, burden, and exceptions

Review the measured operating result, acceptance without material correction, total human review/correction burden, exception rate, ownership, and failure concentration. Ask where work disappeared, where it moved, and which cases remain unresolved.

Minutes 15-20: review controls

Review approvals, source boundaries, prohibited-action attempts, stop behavior, and each material control event. Do not average a control failure into otherwise positive performance.

Minutes 20-25: decide

Select Continue, Change, or Stop. Write the rationale in evidence terms. If the group cannot decide because required evidence is absent, record the item as UNPROVEN and make the evidence-recovery boundary the decision.

Minutes 25-30: assign the next boundary

Before the meeting ends, record the accountable owner and date, next authorized boundary, retained human approvals, named changes, next evidence date, and rollback or stop condition. Read the memo back once so every participant leaves with the same decision.

Keep the decision human-owned and evidence-connected

Practical AI programs work when the agent’s job, evidence, exceptions, controls, and authority stay visible. Microsoft 365, Teams, Copilot Studio, Power Platform, Fabric, and Power BI can support delivery and proof where they fit. They do not replace the operating decision or the person accountable for it.

That accountable pattern connects AI Agents for business operations with the decision layer of Executive Intelligence and practical AI strategy: measured work, named ownership, explicit boundaries, and a repeatable review loop.

Bring one workflow and the evidence you have. Leave with the decision, owner, and next boundary on one page.

Book an AI Operating Review

FAQ

What should an AI agent pilot decision memo include?

Include the workflow and pilot period, original baseline and target, measured result in the same unit, acceptance without material correction, human review burden, exception ownership, control results, evidence quality, the Continue/Change/Stop decision, an accountable owner and date, the next authorized boundary, named changes, the next review date, and a rollback or stop condition.

How do you make a go or no-go decision for an AI agent pilot?

Review measured operating results, quality, human burden, exceptions, controls, and evidence quality together. Choose Continue only within an explicit boundary, Change when a specific design issue can be tested, or Stop when value is insufficient or a risk boundary has been crossed. A failed control can override positive averages.

What if pilot evidence is missing?

Mark the item UNPROVEN. Do not replace missing measurements with estimates or presentation language. Decide whether the gap can be closed in a bounded extension, requires a change to the pilot, or prevents continuation.

Can strong average results outweigh a prohibited action?

No. A prohibited action, failed approval boundary, unowned material exception, or other predefined stop condition can override favorable averages. Record the event, disposition, and authority required before any further test.

Who should own the AI agent pilot decision?

Name one accountable operating leader who can accept the evidence, preserve human approval boundaries, authorize the next bounded step, and stop or roll back the pilot. Contributors may recommend, but the final owner and decision date should be explicit.