Measuring Agentic AI ROI: Cycle Time, Accuracy and Cost per Completed Task
A worked example of the business case for an AI agent: count review and exceptions, separate capacity from cash savings, and measure the whole process.
Read the articleAgentic AIUse-case blueprint
A support-agent blueprint that follows a request through identity, evidence, policy, controlled actions and verified resolution—and explains how to evaluate the whole journey.
ABOUT THE EVIDENCEThis is a use-case blueprint, not a reported customer deployment. The reliability curve is an illustrative probability calculation; benchmark research is cited separately.
Answering a question about a return policy is one task. Completing a permitted return is a longer sequence: identify the customer, locate the order, check the policy, confirm the intended action, update a system and verify the outcome.
A language model can help interpret the request and decide which information is missing. The application still has to establish identity, enforce permissions and keep track of what actually happened.
This article develops an illustrative support workflow around that distinction. The goal is a completed, correct request that can be explained afterward.
Choose a small, named set of request types for a first implementation. Order-status questions, address corrections and return requests may share some infrastructure, but they have different permissions and consequences.
For each request type, specify the records the agent can read, the changes it can propose and the conditions that require a person. Define what counts as successful completion. For an address correction, producing a new address in chat is insufficient; the permitted backend record must be updated and the result confirmed.
It is also useful to distinguish fixed workflows from model-directed planning. Anthropic’s engineering guidance makes this distinction explicit: predefined code paths suit some tasks, while agents choose their own steps and tool use. Complexity should earn its place in the system. [1]
A support application can combine both. The model interprets an ambiguous request and asks for missing details. Deterministic code checks account permissions and action limits. A fixed execution path handles the final database change.
The sequence below is a design blueprint. It does not describe an existing retailer’s policy or a measured SAMVIT Labs deployment.
Authenticate. Bind the request to a verified account. The customer can provide an order reference, but the backend must check that the signed-in identity is entitled to access it.
Gather context. Read the relevant records and retrieve approved policy material. Keep document versions and record timestamps. An answer based on an outdated delivery state can be well written and still lead to the wrong action.
Check policy. Apply the business’s actual rules. Numeric limits, eligibility conditions and approval requirements should be checked in trusted application logic where practical. The model can explain a decision without being the only component enforcing it.
Propose action. Present the consequential change clearly. Ask for confirmation when the task or policy requires it. If there are two possible interpretations, resolve the ambiguity before writing to a system.
Execute once. Submit the authorized change with a stable request identifier. Retries need a way to avoid duplicating a side effect. An application-level idempotency mechanism returns the prior outcome for an already processed request rather than creating another action. Saving the request key and committing the change must be coordinated, including when requests arrive concurrently.
Verify and respond. Read or otherwise confirm the resulting state. A successful HTTP response can mean that a job was accepted, not that every downstream step finished. Tell the user what is confirmed, what remains pending and who owns the next step.
A tool should have a name, an input schema and a result that distinguishes success from rejection or uncertainty. The Model Context Protocol provides a standard way to expose such tools and their schemas. The protocol does not replace the application’s authorization decisions. [2]
For example, prefer a tool that requests a specific, policy-checked operation over a broad interface that can make arbitrary account changes. Keep service credentials outside the model’s prompt. Bind account scope on the server rather than accepting a model-selected tenant as authority.
A small decision record might look like this:
{
"request_id": "example-017",
"action": "request_return",
"status": "needs_approval",
"policy_version": "returns-v3",
"evidence_ids": ["order-example", "delivery-example"]
}
These are illustrative fields. The record helps a reviewer understand the proposal; it is not an instruction to bypass the approval stage. A tool can return a structured rejection that the model then explains to the customer.
OWASP’s Excessive Agency guidance is relevant here: unnecessary functions, broad permissions and excessive autonomy can turn an incorrect model decision into a damaging action. Reduce the available action surface and enforce the remaining permissions in the connected services. [3]
A support benchmark should test more than whether a single reply sounds correct. Did the system use the right account? Did it follow the relevant policy? Did the customer understand the next step? Did the requested change actually occur?
The τ²-Bench research makes the interaction problem concrete. Its telecom setting gives both the simulated user and the agent tools that affect a shared environment. The paper separates reasoning difficulties from communication and coordination difficulties. That is useful evidence for evaluating an end-to-end conversation, although a simulator score does not establish a production completion rate. [4]
Test interruptions as well as successful paths. A user may change their request, a tool may time out, or a human may update the same record during the conversation. The system needs a defined response to each of these events.
Suppose a simplified workflow requires five successful steps. If each step succeeds independently with probability , the probability that every step succeeds is:
Here, is the number of required steps. With and , the complete-workflow probability is about 77.4%. At 99% per step, it becomes about 95.1%.
This is a reason to test complete tasks directly. It is not a substitute for those tests. Retrying a step may improve its success rate while increasing waiting time or the risk of duplicate actions. A common failure in retrieval can affect several later decisions at once.
Per-step reliability is also different from repeated-run consistency. Running the same complete task several times asks whether the system reliably reaches the accepted result across attempts. Both views can uncover problems that one successful demonstration misses.
Keep a compact evaluation table with an explicit denominator for each measure:
| Measure | Definition to agree before the pilot |
|---|---|
| Accepted completion | Eligible cases that reach the correct verified outcome |
| Human handling time | Active review, correction and escalation effort per case |
| Elapsed resolution time | Arrival to verified completion, including waits |
| Unauthorized action attempts | Requests rejected by permission or policy enforcement |
| Reopened cases | Apparently completed cases requiring further work |
| Cost per accepted resolution | Total operating and handling cost divided by accepted completions |
A low handoff rate is useful only when cases are actually resolved. Customer abandonment, silent failures and unsupported requests should remain visible. Record why a handoff occurred and whether it arrived with enough context for the human to continue efficiently.
For a first pilot, a narrow service boundary is an advantage. It lets the team inspect real failure paths, tune the evidence retrieval and decide which actions are appropriate to automate. Expansion should follow evidence that the current scope works.
The resulting system has a clear division of responsibilities: language understanding helps navigate the request; business logic constrains actions; tools change state; verification closes the loop. That division makes improvements easier to measure and failures easier to repair.
Model Context Protocol · 2025-11-25 specification · 2025
Back to textVictor Barres, Honghua Dong, Soham Ray, Xujie Si and Karthik Narasimhan · 2025
Back to text