How Do You Pilot an AI Agent Without Wasting a Quarter?

Pick one job that happens every day, run the agent on it in draft mode for four to six weeks, and measure exactly three numbers: how often the output was accepted without edits, how many minutes a reviewer spent per item, and what each run cost. A pilot scoped that way produces a decision. A pilot scoped as “see if AI can help us” produces a demo and another meeting.

Most agent pilots that fail do not fail because the technology did not work. They fail because nobody agreed in advance what success looked like, so at the end there is a pile of impressive outputs, no baseline to compare them to, and a sponsor who has to make a budget decision on vibes.

Pick the job that bores a senior person daily

The best pilot candidate has four properties, and they are easy to check before you start.

It happens every day or close to it. Volume is what lets you learn something in a month. A job that runs twice a quarter gives you two data points and no idea whether the second one was luck.

It is currently done by someone expensive. Not because their time matters more, but because that is where the business case is obvious enough to survive a finance conversation without a spreadsheet full of assumptions.

Its output is checkable in under a minute. Someone has to look at each result during the pilot. If checking takes as long as doing, you cannot run enough volume to learn anything, and your reviewer will quietly stop checking properly in week two.

It is reversible. Nothing in a pilot should be a one-way door. Draft instead of send, propose instead of post, flag instead of delete.

Inbox triage, lead response drafting, job documentation from field notes and reconciliation flagging all fit that profile. Pricing, contract terms and anything that touches a customer unsupervised do not, and they are not where you start.

Run it in draft mode and keep the gate shut

For the whole pilot, the agent prepares and a person releases. That is not a limitation of the pilot, it is the mechanism that makes the pilot useful.

Draft mode gives you something you cannot get any other way: a record of what the agent would have done, graded by a human, at zero risk. Every item the reviewer accepts without a change is evidence. Every item they rewrite is a specification for the skill you are still missing. By week four you are not arguing about whether to trust it, you are reading a list.

The related question of which actions should stay behind a gate permanently, after the pilot ends, is covered in the admin access answer page. The short version is that money movement, outbound sends, deletion and anything a customer reads stay gated for good. What changes after a pilot is your confidence, not the architecture.

The three numbers

Acceptance rate. Of the items the agent produced, what share went out with no edits at all. Not minor edits. None. Minor edits are a different bucket and you should track it separately, because an output that always needs one small fix is a prompt problem you can solve, while an output that needs restructuring is a scope problem you cannot.

A useful read: if acceptance rate is climbing week over week, the skill is converging and you should extend. If it is flat after three weeks, the job is either harder than it looked or underspecified, and more time will not fix it.

Reviewer minutes per item. The point of the deployment is to give time back, and an agent that produces work needing careful review has moved the work rather than removed it. Measure the real number by asking the reviewer to time one day a week, not by estimating at the end.

Compare it to the baseline you captured before you started. If you did not capture a baseline, capture it now, in the first week, before the agent is involved. The single most common pilot mistake is having no before.

Cost per run. Model usage is billed by the token, so cost moves with how much the agent has to read, not with how valuable the output is. Log it per run from day one. A pilot that does not log cost cannot answer the scaling question, which is the question the whole exercise exists to answer. The mechanics of that line, and what makes it drift, are in what it costs to run an AI agent.

Four to six weeks, and here is why

Two weeks measures novelty. The reviewer is engaged, the volume is unusual, and everyone is paying attention. You learn whether the thing runs at all, which you could have learned in two days.

Four to six weeks covers a full cycle of most operational work: the normal weeks, the end of month, the week somebody is on holiday and the week everything arrives at once. That last category is where you find out whether the agent degrades gracefully or produces confident nonsense under load.

Beyond six weeks you are no longer piloting, you are operating without having made a decision. If you need a quarter, the honest read is usually that the scope was too broad and you should have split it.

What to decide at the end

Three outcomes, and all three are fine.

Scale it. Acceptance rate is high and climbing, reviewer minutes are well below baseline, cost per run is predictable. Open specific gates where the evidence supports it, keep the rest shut, and move to the next job.

Narrow it. The agent is excellent on two thirds of the volume and poor on the rest. This is the most common result and the most useful one. Route the two thirds to the agent, route the rest to a person, and stop trying to make one skill cover both.

Stop it. Acceptance rate is flat and low, or reviewer time did not move. Say so plainly and write down why, because the reason is usually transferable: the inputs were not available in a form the agent could read, or the job needed context that lives in someone’s head and was never written down. Both of those are fixable later and neither is a verdict on agents in general.

The part people skip

Write the pilot definition down before the first run. One page: the job, the baseline numbers, the three measurements, the review process, the duration and what each of the three outcomes would mean. Have the sponsor and the reviewer both agree to it.

It takes an hour and it is the difference between a pilot that ends in a decision and one that ends in a meeting about whether it worked. If you are still deciding whether the job in question even needs an agent rather than a simpler fixed pipeline, run the flowchart test first. A workflow you can draw in advance is cheaper, more reliable and does not need a pilot at all.

When you are ready to scope one properly, a discovery call is 30 minutes and ends with the job picked and the three numbers defined.