How to test whether an AI agent falsely reports success after a tool fails

Fix a failed-tool observation and keep the evidence needed for completion outside the agent's context. Score its final report for unsupported success and invented detail, while also checking useful recovery. Pair blocked-task cases with successful-tool controls and full-workflow evaluations before setting a release gate.

A browser timeout does not establish that a page was verified. A missing attachment does not support a summary of that file. A crashed test runner does not support “tests pass.” The FTA study separates what was requested, attempted, and observed so the final report can be tested on its own.

Why isolate the final report?

An end-to-end agent test combines tool choice, execution, retries, environment changes, and final communication. FTA gives the model a request and deterministic failed-tool trace. Evaluator-side records retain the absent completion evidence, a feasible recovery action, and safe partial help. That makes a false success claim identifiable without confusing it with tool-selection failure.

This diagnostic slice does not replace full-run evaluation. OpenAI's agent-evals guide describes scoring traces of model calls, tool calls, guardrails, and handoffs against structured criteria. Keep that broader view beside the focused reporting test.

Build one blocked-task case

FTA's 100 synthetic tasks cover unavailable retrieval, missing attachments, failed execution, permission denial, and stale data, with neutral and user-pressure conditions. Use the failure families relevant to your tools. Keep the observation identical across response-policy variants, and do not put the evaluator's required-evidence key in the agent prompt.

Score fidelity and usefulness separately

A passing blocked-task answer might say: “I could not run the tests because the runner failed to start. I have no test result, so I cannot say they pass. I can retry after the environment is fixed.” This is an illustration, not a benchmark quote. Hedging does not create an observation; explaining a next step is not a success claim.

Test an evidence-formatted response policy

FTA compared baseline, a plain transparency instruction, and a structured response with STATUS, EVIDENCE, LIMITATION, and NEXT ACTION fields on the same failed observations. In its six-model, 3,600-response synthetic blocked-task evaluation, pooled descriptive false-success rates were 22.8%, 9.3%, and 0.8%; useful-response rates were 74.9%, 89.2%, and 98.8%, respectively.

The additional three models were evaluated after the original cohort was analyzed, making the six-model pool descriptive. The structured policy changes wording, constraints, and format together; the experiment does not isolate which part mattered. These figures do not predict an improvement for your agent.

Add controls before a release gate

1. Matched successful-tool cases: check that real completion evidence produces a justified success report, not reflexive refusal. FTA includes failed prerequisites only and calls for these controls.
2. Full-run trace cases: check tool choice, failure detection, permitted recovery, and whether the report matches the actual run.
3. Observed-outcome cases: for state-changing work, compare the claim with the persisted result. See evaluating stateful agent outcomes.

Keep reporting fidelity beside normal task success. Root-cause analysis explains why a tool failed; this test asks whether the user was then told the truth. Set local criteria through the agent release-threshold process, using case severity and your own baseline.

What this research does not establish

FTA does not measure tool selection, autonomous retry, partial success, contradictory evidence, multi-agent runs, or production incident frequency. Cases are synthetic, English, predominantly one-step, and lack real-world trace validation. Human scoring did not report independent double-annotation or inter-annotator agreement; response formatting could make the policy condition inferable to scorers.

Make the evidence needed for success explicit, then test whether the final answer stays within what the agent actually observed after a tool fails. Keep successful-tool and end-to-end checks beside this diagnostic test so honesty does not mean never completing work.