How to tell whether an agent fault-injection test exposed the fault

An agent's use of a bad tool result is not, by itself, evidence that it ignored a contradiction. For every injected fault, define the observation that would refute it, prove the agent could legally obtain that observation before first dependent use, then reconstruct whether it actually arrived in that run. Only timely, model-visible counterevidence supports the stronger claim that the agent relied on the fault despite seeing a contradiction.

Start with a fault contract, not a corrupted response

Suppose a reservation-update tool returns success although the write was suppressed. An authoritative read-back of the same reservation can contradict that acknowledgment. But if it arrives after the agent tells the user the change was made, it cannot show that the agent ignored evidence before making the claim. SSCBench uses this false-write example to distinguish the evidence conditions behind superficially similar failures.

This worksheet is a Bonfire adaptation of the paper's fault-contract definitions, not a published benchmark template. A refutation must be authoritative, about the same entity and scope, independent of the mutation, and sufficient to contradict it from the agent's available information. A merely related product record, or knowledge held only by the evaluator, does not qualify.

Separate the opportunity from what happened

A case is pre-use revealable if at least one legal continuation from the fixed injection context could expose valid counterevidence before first dependent use. The needed tool arguments must be obtainable by the agent under its actual permissions. This admits the case for evaluation; it does not mean the evaluated agent followed that path.

Classify the realized trace by when valid evidence arrived relative to first dependent use:

• Pre-use: valid counterevidence arrived before that use.
• Late: it arrived only afterward.
• Absent: it did not appear in the observed run.
• Undecidable or no use: retain separate labels rather than silently counting either as absent or successful.

The paper uses first dependent use as the common timing anchor. First faulty reliance is a separate diagnosis within adopted runs; a verification call alone is not necessarily reliance. In the study, among 44 adopted runs with eventual counterevidence, 17 received it before first use and 27 afterward. Those counts describe the tested configurations, not deployed agents.

Report each claim with its own denominator

A pre-use revealable case can still produce a late or absent-evidence run. Preserve the complete model-visible trace, tool observations, and grading labels. OpenAI's agent-evals guide describes traces across model calls, tools, guardrails, and handoffs and structured grading. It supports retaining and scoring evidence, but does not validate SSCBench's labels or rates.

Review temporal labels before using them as a gate

Record fault delivery, valid evidence arrival, first dependent use, and—if adoption occurred—first faulty reliance. For a refutation assembled from several observations, arrival occurs when the last required piece becomes model-visible. Check entity matching and contradiction semantics, not timestamps alone. SSCBench reports that automated trajectory analysis did not reliably recover the exact first faulty-reliance event; manually adjudicate ambiguous cases or mark them undecidable.

Keep prevention, later stance revision, and repair of external state separate. A corrected final answer does not prove an earlier API write or user-visible effect was undone. For the adjacent communication failure, see testing false success after tool failure; for state-changing outcomes, see evaluating stateful agent outcomes.

What the study can and cannot establish

SSCBench evaluated four fault operators and five agent configurations across 1,191 faulted executions in two tau-bench environments. The protocol applies when a task-authoritative refutation and its relation to downstream use can be specified. It does not cover every failure or establish universal model rankings, production incident frequency, or that a contradiction guarantees correction. The practical test is modest: define the refutation, preserve what the agent actually saw, and name the evidence condition behind each adoption claim.