Agentic AI: Towards reliable AI agents

1. Introduction

A capable agent can still be an unsafe system.

In July 2025, Replit Agent deleted data from an application database during development; because development and production were connected, the change affected the live application. Replit later said a checkpoint rollback restored the database. In February 2024, British Columbia's Civil Resolution Tribunal held Air Canada liable after its website chatbot provided misleading bereavement-fare guidance. In a Washington Post test, OpenAI's Operator purchased eggs for $31.43 without the requested confirmation. New York City's MyCity chatbot repeatedly gave businesses incorrect—and, in separate repeat tests, inconsistent—legal guidance.

These cases span agents and chatbots with different architectures, but they expose the same evaluation gap. Benchmarks often ask whether a system can complete a task once. Deployment asks whether it behaves correctly, consistently, robustly, predictably, and safely across repeated real interactions.

Agents are not pure functions. They are systems assembled from a model, prompt, memory, tools, policies, permissions, orchestration, and external state. Their outputs may vary even when the visible inputs do not. A single green run tells us little about their behavior distribution.

Two studies provide useful foundations for measuring that gap:

• τ-bench introduced a reproducible tool-agent-user environment and \(\mathrm{pass}^k\), a measure of repeated success rather than best-of-\(k\) success.
• Towards a Science of AI Agent Reliability, accepted at ICML 2026, operationalized four dimensions of reliability: consistency, robustness, predictability, and operational safety.

2. τ-bench: task success is not enough

A deployed enterprise agent faces conditions that classic one-shot benchmarks rarely model:

• users reveal intent incrementally and imprecisely;
• tools expose partial and changing state;
• policy documents constrain permitted actions;
• some actions modify persistent records; and
• millions of similar requests must receive similar treatment.

Benchmarks are the feedback loop of agent engineering. Without a reproducible evaluation environment, failures remain anecdotes, prompt changes are hard to compare, model upgrades introduce silent regressions, and teams optimize whichever aggregate score is easiest to observe.

The missing question is not only, Can it succeed? It is, Will it behave reliably in production?

2.1. Environment design

The original τ-bench models customer-service agents in two domains:

• τ-retail: 115 tasks, 500 users, 50 products, 1,000 orders, and 15 API tools.
• τ-airline: 50 tasks, 500 users, 300 flights, 2,000 reservations, and 13 tools. Airline tasks are harder because rules depend on fare class, cabin, membership, timing, and payment method.

Later work extended the benchmark family. τ²-bench added telecom and dual-control tasks in which both the agent and simulated user can call tools. Subsequent τ-bench work added corrected core tasks, a banking knowledge domain, and full-duplex voice evaluation. Because the benchmark and its leaderboard continue to evolve, historical and current results should not be mixed without an explicit date and benchmark version.

Each environment contains a database, API tools, a natural-language policy, simulated users, and tasks with annotated goal states. The agent cannot see the user's hidden goal or the database directly; it must infer intent through dialogue and inspect or modify state through tools. This matters because production agent work is partially observable. The task is not merely selecting the right function. The agent must ask for missing information, retrieve state, apply business rules, call tools with correct arguments, and communicate the result.

Simulated user. An LLM plays the customer. The simulated user receives a hidden instruction fixing identity, intent, and preferences—for example, “You are Mia Li. You want to fly to San Francisco instead of Los Angeles. You are concise.” Sampling creates varied conversations for the same underlying task. That makes repeated trials affordable, which is the raw material for reliability measurement.

Outcome-based evaluation. A task is successful only when the final database state matches the annotated goal and the required information appears in the agent's messages:

The scorer checks the endpoint rather than matching a reference trajectory. \(r_{\text{action}}\) compares the final database state; \(r_{\text{output}}\) checks required response substrings. Both are deterministic and inexpensive to recompute, which makes repeated sampling—and reporting \(\mathrm{pass}^k\)—practical.

Because the terms multiply, a run that gets the database right but fails to tell the customer scores the same zero as a run that says the right thing over the wrong database. Both the state and the message must be correct.

The cost moves to annotation. Each task needs a hand-labeled goal state, and a task with several valid endings must represent every acceptable endpoint or risk scoring correct runs as failures. The τ-bench authors ran each retail task more than 40 times and inspected tasks with zero or unusually low success for ambiguity, annotation errors, or infeasibility.

Endpoint grading is necessary but not sufficient. A run can reach the expected final state while skipping a required process step—for example, issuing a refund without confirmation. Credible evaluation therefore needs both endpoint assertions and trajectory checks for policy invariants, authorization, and consequential actions.

The evaluator itself also needs validation. The current reliability study reports—citing a later audit—that 24 of the original 50 airline tasks contained flawed labels or ambiguous specifications; it therefore uses a clean 26-task subset. Benchmark tasks, simulators, and graders are software, and should be versioned, tested, and audited like the agent under evaluation.

2.2. pass@k and pass^k

The τ-bench paper introduces \(\mathrm{pass}^k\) as a complement to the commonly used pass@k. The metrics answer different questions.

pass@k measures the probability that at least one of \(k\) attempts succeeds. It suits code generation, where a model can produce several candidates and a test suite selects a correct one. More attempts improve the odds of finding one.

\(\mathrm{pass}^k\) measures the probability that all \(k\) attempts succeed. It suits customer-facing and business-critical deployment. A customer receives one execution, not a shortlist. Repeating the task must keep producing the correct result.

For task \(t\), evaluated \(n_t\) times with \(c_t\) successful runs, the estimators are:

The difference is important:

• pass@k increases as more attempts are allowed, because only one attempt must succeed.
• \(\mathrm{pass}^k\) decreases as more successful repetitions are required, because every attempt must succeed.

For a single task with a stationary 90% success probability and independent runs:

The agent looks strong under single-run accuracy but has less than a 50% chance of completing that task correctly eight times in a row. The power calculation is only an illustration: heterogeneous tasks and correlated failures violate its assumptions. Empirical \(\mathrm{pass}^k\) should be reported with the number of trials and uncertainty rather than extrapolated from one average.

pass@k rises as more attempts are allowed; \(\mathrm{pass}^k\) falls as repeated correctness is demanded. The gap between the curves is the difference between search capability and service reliability.

2.3 What τ-bench found

In the original 2024 study, the best reported configuration—GPT-4o with native function calling—reached 61.2% \(\mathrm{pass}^1\) on retail and 35.2% on airline, or 48.2% averaged across the two domains. Retail \(\mathrm{pass}^8\) fell below 25%. A task the agent could solve once, it often could not solve every time.

Manual analysis of 36 failed retail runs attributed approximately:

• 55.6% to wrong information or tool arguments;
• 25% to wrong policy decisions; and
• 19.4% to incomplete multi-step execution.

These categories suggest candidate controls—schema and referential validation, policy-boundary tests, and explicit completion checks—but the study did not test whether those interventions reduced the failures.

A policy ablation was especially informative. Researchers reran the agent without the policy document. In that configuration, performance fell by 4.4 percentage points on retail and 22.4 points on airline. The authors suggest that retail rules were simpler and closer to common sense, while airline tasks depended more heavily on fare class, cabin, timing, and payment rules. The smaller retail change does not show that policy was unimportant; it may also indicate that the agent was not using the supplied policy effectively.

Those figures describe the original 2024 benchmark, whose tasks are now labeled outdated by its maintainers. They remain historically useful when presented as such. The durable result is not a particular leaderboard rank: it is that average task success can hide severe run-to-run inconsistency, and that reliability varies substantially by domain and policy complexity.

3. Towards a Science of AI Agent Reliability

3.1 Capability outran reliability

Rabanser, Kapoor, Kirgis, Liu, Utpala, and Narayanan evaluated 15 model configurations spanning release dates from March 2024 through May 2026. They used benchmark-specific agent scaffolds on 165 GAIA validation tasks and the 26 verified τ-airline tasks, running each task five times.

Across the sampled releases, GAIA accuracy rose by 0.23 per year while aggregate reliability rose by 0.03 per year. On τ-airline, the corresponding slopes were 0.22 and 0.09. These are descriptive regressions over particular model releases, benchmarks, and scaffolds—not a causal test of scaling. They nevertheless show that capability gains did not transfer uniformly to reliability.

Accuracy alone cannot reveal this difference. Consider two agents with equal 90% accuracy. One fails on the same identifiable 10% of tasks every time; those tasks can potentially route to a human. The other fails randomly across every task class. Their averages match, but only the first supports predictable selective automation.

The framework draws on reliability engineering, forecasting, risk analysis, and safety engineering. It separates reliability into four dimensions:

• consistency;
• robustness;
• predictability; and
• operational safety.

The paper's aggregate reliability score combines consistency, robustness, and predictability. Safety is reported separately because low-frequency, high-severity events should not be averaged away.

3.2 Consistency: repeated runs still diverge

Consistency asks whether the same task under the same nominal conditions produces the same result.

Each task was run \(K=5\) times. The authors used temperature zero where the API exposed it; reasoning models retained provider defaults when temperature was not configurable. Outputs can still vary because decoding temperature is not the only source of nondeterminism in a distributed inference and tool-execution stack. Temperature zero reduces one source of variation; it does not guarantee determinism.

If \(\hat{p}_t\) is task \(t\)'s empirical success rate, the current paper defines outcome consistency as:

The score is 1 when a task always succeeds or always fails and 0 when it succeeds half the time. It therefore measures repeatability, not capability, and must be reported alongside accuracy. \(\mathrm{pass}^k\) does not make the same separation: an agent that fails every run has \(\mathrm{pass}^k=0\), even though its outcome is perfectly consistent.

To make the corrected formula concrete, take three tasks, each run \(K=5\) times:

Tasks A and B are capability opposites, yet both are perfectly consistent because all five runs agree. Task C splits three-to-two and receives only 0.04. Averaging the three terms gives \(C_{\text{out}}=(1+1+0.04)/3\approx0.68\). Unlike the formula in an earlier preprint, the current metric decreases smoothly as the empirical success rate approaches 0.5; it is not merely the fraction of tasks with unanimous outcomes. With \(K=5\), an empirical rate of exactly 0.5 is impossible, so 0.04 is the lowest observable per-task score.

Outcome consistency does not show whether an agent follows the same process across runs. Two runs may both succeed while taking materially different paths. For enterprise agents, that difference matters because action order affects safety, recoverability, and auditability.

The paper therefore evaluates trajectory consistency at two levels. It compares action-type distributions using Jensen–Shannon divergence and action ordering using normalized Levenshtein similarity. The results show a common pattern: agents are generally more consistent in what actions they use than in when they use them.

That distinction matters in transactional systems. An agent may check inventory before charging in one run and charge first in another. Both paths may succeed under normal conditions, but an interruption can leave different system states. One path is recoverable; the other may create an incorrect charge, inconsistent records, or a harder audit trail.

In several within-family comparisons, smaller models matched or exceeded larger models on consistency. The authors hypothesize that larger models may exploit more valid solution paths, but the evaluation does not establish that mechanism. Multiple valid trajectories are not inherently undesirable; variation matters when it changes consequential action order, recovery behavior, cost, or auditability.

3.3 Robustness: what happens when variables change

Robustness measures whether performance survives changed conditions. For positive baseline accuracy, the paper compares perturbed accuracy with baseline accuracy:

The ratio is capped at 1 because it measures retained performance rather than improvement. Baseline and perturbed accuracy should be reported beside it: a low-accuracy agent can retain a high fraction of already-poor performance, while the cap hides improvements.

The paper tests three categories of perturbation:

• a fault suite, injected on each wrapped API or tool call with probability 0.2, including timeouts, HTTP 500 and 429 responses, network errors, and partial, malformed, or empty responses;
• controlled environment variations; and
• five semantically equivalent prompt rewrites per task.

For an illustrative 10-task evaluation, suppose baseline accuracy is 0.90:

The environment variation happens to improve accuracy, producing a raw ratio of \(1.00/0.90=1.11\), but the cap holds \(R\) at 1. The metric measures retained performance, not a lucky gain. Paraphrasing removes nearly half the baseline performance in this example.

In this suite, fault and environment robustness showed ceiling effects, while prompt robustness varied more, especially on GAIA. The authors caution that these perturbations cover only a narrow slice of deployment change and do not include every form of schema migration, API-version change, or interface redesign.

Prompt robustness remains important because users rarely reproduce benchmark wording. A system may recover from a tool timeout yet fail when “cancel my subscription” becomes “please end my plan.” That example is hypothetical, but the deployment lesson is direct: robustness tests should represent the language, tools, policies, and state changes the production system will actually encounter.

3.4 Predictability: predicting the likelihood of failure

Predictability asks whether the agent can identify when it is likely to fail. After each run, the researchers showed the model its complete trajectory and requested a confidence assessment from 0 to 100, normalized to \(c_i \in [0,1]\). The metrics therefore evaluate elicited post-hoc self-assessment, not a model-internal probability exposed during execution.

The paper evaluates three related properties.

Calibration

Expected Calibration Error groups predictions into confidence bins and compares mean confidence with observed success in each bin. Suppose two bins hold ten tasks each. The first averages confidence 0.75 but succeeds 60% of the time, a gap of 0.15. The second averages confidence 0.25 and succeeds 30% of the time, a gap of 0.05. Then \(\mathrm{ECE}=0.5(0.15)+0.5(0.05)=0.10\) and \(P_{\text{cal}}=0.90\).

Discrimination

AUROC asks whether a successful run receives higher confidence than a failed run, with ties receiving half credit. A value of 1 is perfect ranking; 0.5 is random. For successes with confidences \(\{0.9,0.8,0.6\}\) and failures with \(\{0.7,0.4,0.3\}\), the success ranks higher in eight of nine pairs, so \(P_{\mathrm{AUROC}}=8/9\approx0.89\).

Brier score

The Brier score evaluates the squared difference between confidence and outcome for every run. A run assigned confidence 0.9 that succeeds contributes \((0.9-1)^2=0.01\); one assigned 0.7 that fails contributes \((0.7-0)^2=0.49\). Across those two runs, \(P_{\mathrm{Brier}}=1-(0.01+0.49)/2=0.75\). The paper reports one minus the usual Brier loss so that higher values are better.

Calibration and discrimination can improve independently. A model may be well calibrated across a population yet fail to identify which individual cases are risky. In the study, calibration improved among recent models, while discrimination generally improved on τ-airline but stagnated or worsened for some recent models on GAIA. The evidence does not support a benchmark-independent claim that failure detection either improved or failed to improve.

This distinction determines whether confidence can support selective automation. Raw model self-confidence should not control routing or release gates until it is independently calibrated on the deployment distribution. In production, combine it with deterministic risk signals such as missing evidence, conflicting policy, unsupported intent, unusual tool results, and the value or irreversibility of a proposed action.

3.5 Safety: reported separately, deliberately

The paper uses safety in a narrow operational sense: violations of task policies and the severity of those violations. It does not evaluate adversarial attacks, alignment, broad social harms, or the complete security of an agent system.

It measures two parts:

• Compliance, \(S_{\text{comp}}\), is the fraction of runs without a policy violation, so \(\Pr(\text{violation})=1-S_{\text{comp}}\).
• Harm severity, \(S_{\text{harm}}\), summarizes the maximum identified severity for violating tasks. Violations are bucketed as low (0.25), medium (0.5), or high (1.0), and \(\mathbb{E}[\text{severity}\mid\text{violation}]=1-S_{\text{harm}}\).

The paper operationalizes safety as:

Higher values are safer. Safety is excluded from aggregate reliability because tail events dominate it. An agent that is safe in 99% of runs but catastrophic in 1% should not earn a reassuring score through averaging.

The expected-harm score illustrates its own limitation:

The rare-catastrophic agent receives the higher score because the catastrophic event is less frequent. A single expected-harm number cannot represent an unacceptable tail, so severity-specific release gates must remain visible.

GPT-4o judges whether a trajectory violates policy and assigns severity. That makes evaluator error a study limitation: a model-based safety score is evidence, not ground truth. The most frequent violation category in the τ-airline evaluation was financial inaccuracy, including incorrect charges and refunds.

Safety engineering in aviation and other high-consequence fields similarly uses severity-aware objectives rather than a good mean outcome. The analogy is useful, but quantitative targets do not transfer directly from aircraft certification to AI-agent deployment.

3.6 Reinterpreting the four opening incidents

The four dimensions help classify the opening incidents, but they do not establish root causes from public reports alone.

• Replit — primarily an authorization and containment failure with potentially high severity. The agent could mutate a production-connected database; the documented incident does not demonstrate prompt-paraphrase sensitivity, and Replit says rollback restored the data.
• Air Canada — a grounding and deployment-accountability failure: a public interface gave guidance contradicted by the authoritative policy page. Because the tribunal record does not reveal the chatbot architecture, it is not clean evidence about LLM-agent reliability.
• Operator — a confirmation-compliance failure visible in the action sequence: it completed a purchase although the user had requested confirmation first.
• NYC MyCity — factual-accuracy and outcome-consistency failures. Investigators observed wrong legal guidance and different answers in repeat testing; calibration was not numerically measured.

A common objection is that more capable models will become reliable automatically. Perfect endpoint accuracy would make outcome consistency trivial, but on a finite benchmark it would not by itself establish trajectory or resource consistency, robustness beyond the tested distribution, confidence quality, or operational safety. Below that limit, two agents with equal success rates can differ sharply across these dimensions. In this study, newer models' capability gains were not matched by comparable gains in every reliability dimension. That is descriptive evidence across releases, not a causal proof that scaling cannot improve reliability.

4. Practical use: Integrating reliability into the agent development lifecycle

Metrics only matter if they change how we build. The agent development lifecycle (ADLC) is the agent-specific counterpart to the familiar SDLC, and it runs in six steps. This lifecycle is an engineering synthesis from the benchmark evidence, not a procedure experimentally validated by either paper.

1. Define the operating envelope. Specify supported intents, users, languages, workflows, tools, policies, permissions, external conditions, and failure modes. Reliability is always relative to this versioned envelope.

2. Build a representative evaluation environment. Use production-faithful API simulators, isolated database copies, real policy documents, controlled state, simulated users, and explicit acceptable endpoints. Never point destructive evaluation traffic at production. If several outcomes are valid, the grader must recognize all of them.

3. Run repeated and perturbed tests. A single success is weak evidence. Report \(\mathrm{pass}^1\) and \(\mathrm{pass}^k\) with trial counts and uncertainty. Test paraphrases, tool faults, environment changes, policy boundaries, concurrent updates, and interrupted workflows. Agent CI requires repeated runs, not one green execution.

4. Create a context-specific reliability profile. Report consistency, robustness, and predictability separately. Keep operational safety visible through violation-frequency and severity-specific gates rather than averaging it away. The acceptable profile depends on the application: variation may help brainstorming but is dangerous in transactional workflows.

5. Enforce risk-specific release gates and runtime controls. Human-reviewed systems can tolerate more uncertainty than autonomous ones. As consequence and autonomy rise, require stronger evidence and tighter controls: least-privilege credentials, separate read and mutating tools, deterministic validation, confirmation before consequential actions, idempotency keys, transaction limits, postcondition checks, rollback, and a kill switch. Evaluation estimates when to trust the agent; containment limits the cost when that estimate is wrong.

6. Turn production failures into regression tests. Add every incident, near miss, and human overturn to the evaluation suite. Preserve versioned traces of prompts, policies, tool calls, validated arguments, approvals, mutations, retries, outcomes, and recovery—subject to privacy and retention controls. Requalify the system when a model, prompt, tool, policy, permission, evaluator, or traffic distribution changes. Qualification in January does not guarantee qualification in June.

5. Open problems

Three problems remain open.

1. Benchmark decay. Fixed test sets invite contamination and shortcut exploitation. Generative or parameterized environments can vary prompts, schemas, state, and faults while preserving task semantics.

2. Multi-agent attribution. Multi-agent systems need better reliability metrics and failure-attribution methods because errors propagate across component boundaries.

3. Evaluator reliability. Safety and trajectory evaluation often depend on LLM judges. Those judges introduce a second model whose errors require independent validation. Human-validated and judge-free safety measurements remain important research problems.

6. Conclusion

The key question is no longer only, “How often does the agent succeed?” It is, “How consistently, robustly, predictably, and safely does it behave?”

τ-bench made repeated task success measurable. The reliability framework separated reliability into diagnostic dimensions. The development lifecycle connects those measurements to testing, release, monitoring, and containment.

The Replit agent could write code, yet the system around it did not adequately contain its actions. Closing that gap requires an engineering discipline for measuring behavior, limiting damage, and improving reliability—not another model release alone.

References

1. Yao, S., et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Paper

2. Barres, V., et al. (2026). τ²-bench: Evaluating Conversational Agents in a Dual-Control Environment. International Conference on Machine Learning (ICML 2026). Paper

3. Rabanser, S., et al. (2026). Towards a Science of AI Agent Reliability. International Conference on Machine Learning (ICML 2026). Paper

4. Cuadron, A., et al. (2026). SABER: Small Actions, Big Errors—Safeguarding Mutating Steps in LLM Agents. ICLR 2026 Workshop paper. Paper