How to measure an AI operator pilot without confusing speed with reliability

Start with the decision the pilot must support

A useful pilot is not designed to prove that an interface can move quickly. It should establish whether Vigilo can observe, prepare or execute a precise workflow within a controlled scope. The decision might be to authorize the creation of certain orders, add an inbound channel, cover a customer group or retain human validation for a specific exception.

That target decision determines what evidence must be collected. Without it, record volumes and average processing times can look impressive while remaining operationally meaningless. Before testing begins, the business and technical owners therefore define the scope, authorization level, prohibited risks and evidence required to proceed.

Define the unit of measurement: record, decision and outcome

The number of automated clicks does not describe work quality. The useful unit is a complete loop: an event arrives, it is linked to a record, its intent is classified, an authorized rule leads to a decision, an action is prepared or executed, and the resulting state is checked.

Every observation must remain reviewable with its sources and rule version. For example, an email amendment received after an EDI order counts as correctly processed only when it is linked to the right record, applied to the right field and visible in the final TMS state.

1. 01
   Event

   What information arrived, through which channel and at what time?

2. 02
   Linkage

   Which record was the event linked to, and how ambiguous was the match?

3. 03
   Decision

   Which rule, source and approval explain the action or stop?

4. 04
   Action

   What was prepared or executed, in which tool and with which permissions?

5. 05
   Outcome

   What state was actually observed after the action?

6. 06
   Human follow-up

   Who approved, corrected, rejected or completed the processing?

Keep measurement families separate

These families should not be collapsed into a single score. An average may conceal weak record linkage or poor exception handling. The dashboard retains numerators, denominators and correction reasons so the team can see what is improving and what must remain prohibited.

Do not confuse coverage, quality and autonomy

Coverage is the share of events or records included in the evaluated scope. Quality is the correctness of processing on those cases. Autonomy is the share of decisions and actions authorized without intervention. Broad coverage with insufficient quality does not justify greater autonomy; high quality on a handful of routine cases does not prove broad workflow coverage.

Two types of error must also remain distinct. A false approval allows a decision or action that should have been stopped. A false stop unnecessarily returns a mastered case to the team. Their operational costs differ, so thresholds and review methods should reflect workflow risk rather than a universal average.

Build a sample that resembles real work

Cases should not be mixed indiscriminately into one rate. They are grouped by channel, customer, request type, risk level and workflow version. This segmentation shows whether an improvement reflects a better method or merely a larger proportion of easy records.

When a record is used as ground truth, the expected decision must be reviewed by an authorized person. Disagreement between reviewers is retained: it may reveal an ill-defined rule rather than a system error.

• frequent, well-documented cases;
• incomplete or contradictory requests and information received through several channels;
• amendments, cancellations and documents received after record creation;
• similar names, close references and other difficult linkage cases;
• customer-specific rules and sensitive decisions;
• outages, insufficient permissions or changes to the target interface;
• new cases that must explicitly remain out of scope.

Test adaptability without promising magical compatibility

Vigilo does not require every company to use the same TMS, screen or operating method. Its architecture is intended to adapt sources, rules, permissions, controls and the action mechanism to the real workflow. That adaptability must still be tested; it cannot be inferred from a list of software products labelled compatible.

The pilot documents what remains common — event, record, decision, action and verification — and what depends on the company: fields, screens, rules, exceptions and responsibilities. When an interface or procedure changes, regression tests verify that authorized paths still work and all other paths stop safely.

• time and work required to configure a new flow;
• share of rules reused and share that is genuinely specific;
• behavior after a screen, field or procedure changes;
• ability to use an API when suitable and the user interface when necessary;
• consistent evidence and controls regardless of the target system.

Progress through authorization levels

Progression is never automatic. Business owners confirm decision quality; technical owners validate access, robustness and traceability; risk owners define the situations that remain human.

1. 01
   Observe

   Vigilo reconstructs the record and its decisions without changing any system. Differences from human processing are classified.

2. 02
   Prepare

   Vigilo proposes the action and values; a person reviews them before execution. Corrections become evidence.

3. 03
   Execute within scope

   Only approved actions and rules are authorized. The resulting state in the system is systematically checked.

4. 04
   Expand or maintain

   The team decides whether to add cases, retain supervision or remove an authorization based on segmented results.

Evidence required to reach a conclusion

Time savings and avoided costs may be tracked, but they remain outcomes of the observed scope. They must state the period, volume, calculation method and remaining human work. Without those elements, they prove neither reliability nor a repeatable result.

• definition of scope, case populations and exclusions;
• reference set and human validation method;
• sources, rules and versions used for every sampled decision;
• proposed action, action actually executed and state observed after verification;
• corrections, rejections, escalations and their reasons;
• incidents, system changes and regression test outcomes;
• final decision: expand, correct, retain under supervision or stop.

A template to complete before measurement begins

This template can be copied into the pilot record. It should be completed before results are analysed, then versioned whenever the scope, rules or action method changes. An item that is not yet known should be marked “to be confirmed”, rather than replaced by an unstated assumption.

Turn a case list into a reproducible sample

A sample that represents the scope is not necessarily a uniform random sample. Frequent cases can measure day-to-day operation, while dedicated strata are needed to observe rare but high-risk situations. The selection method must be written before testing so that the sample does not retain only easy or successful records.

There is no universal required sample size. It depends on the actual diversity of the flow and the decision the pilot is meant to support. If an important stratum contains too few observations, the result for that stratum remains inconclusive; it should not be quietly merged with routine cases or presented as validated.

Compare before and after on a consistent basis

The comparison covers all the work required to leave a record in the expected state. The baseline includes reading, searching, data entry, checks, follow-ups and rework. The pilot includes system observation, human review, corrections, escalations, technical recovery and outcome verification. Automated execution time alone therefore measures neither reliability nor the work that remains.

Both periods must use the same definitions. If the case mix, service level, system, team or business rule changes, the difference should be segmented or marked as not directly comparable. Faster processing never offsets a false approval, an incorrect action or missing evidence.

Read excluded cases without flattering the result

Every exclusion receives a dated reason and status. A case that genuinely falls outside the stated scope does not test quality within that scope, but it still matters when calculating coverage and estimating human workload. By contrast, a case abandoned because the system, access method or control failed belongs in the technical assessment. Removing it would hide the exact weakness the pilot is supposed to uncover.

At a minimum, the report therefore provides four counts for each stratum: observed population, in-scope cases, evaluable cases and correctly processed cases. It also reports the volume handed back to the team and the associated recovery time. This prevents an excellent rate among accepted cases from concealing low coverage or substantial remaining work.

Once technical validation is established, the guide to calculating transport automation ROI can assess investment, operating costs and benefits actually observed as a separate question. A reliable pilot is not proof that a project will be profitable, and an ROI scenario does not establish technical reliability.