Which AI agent caused the failure? Audit the handoff

To find where a multi-agent run failed, trace each consequential decision to the information it used, then check whether the receiving agent preserved, distorted, ignored, or repaired that information. A dependency shows provenance, not responsibility. Validate any attribution against the whole trace, and leave it indeterminate when evidence is missing.

In the paper's motivating example, a tool result correctly gives Braintree and Honolulu, but a downstream agent reads it as Honolulu and Quincy; another carries the mistake into the final answer. Blaming the tool or tagging only the final speaker would miss the receiving decision where the claim changed. This is a paper example, not a Bonfire incident.

Keep an intact run record

Preserve the task request, actor IDs, messages, tool calls and results, handoffs, and final artifact together. Mark missing or shortened segments. OpenAI's agent-evaluation guidance describes traces as end-to-end records of model calls, tool calls, guardrails, and handoffs, and recommends inspecting representative traces before formalizing graders. BLADE likewise depends on recorded observations; truncation can omit relevant evidence.

For each suspected handoff, record four fields. This is a proposed incident-review record adapted from BLADE, not a schema validated by the study.

Separate provenance from responsibility

BLADE reconstructs dependencies, selects source–decision handoffs, audits how sources were used against task requirements, then makes a global diagnosis. The selected handoffs do not restrict which event can receive the final attribution. A suspicious local handoff may not matter to the failed outcome; an earlier error may be corrected later.

1. Map plausible dependencies. Identify sources for consequential decisions, including nonadjacent handoffs.
2. Audit source use. Compare the actual source with the receiver's claim or action and the task requirement. Record uncertainty instead of forcing a binary fault label.
3. Test the whole-run explanation. Ask whether correcting the suspected step could plausibly change the outcome. Look for later repairs and independent failures.
4. Make a bounded disposition. Name a likely error-introducing step only when the trace supports it; otherwise request human review or better instrumentation.

The benchmark's counterfactual decisive-step framing is a useful review question, not a guarantee that a real incident has one knowable cause. For the complementary single-trajectory view, see agent failure root-cause analysis.

What the study measured

The authors evaluated BLADE on 1,228 failed trajectories across five Who&When and Who&When Pro benchmark subsets, including natural and injected failures. Eight attribution strategies were compared under a shared Jev judgment interface. BLADE reported 57.6% macro-average exact-step accuracy versus 52.0% for the strongest step baseline, and 66.8% agent accuracy versus 64.0% for the strongest agent baseline. These are equally weighted benchmark agreement rates, not production incident accuracy.

In an ablation across four subsets, adding handoff-use findings to dependency evidence improved exact-step accuracy by 2.4–3.4 percentage points. Dependency-guided selection audited 13–23% as many handoffs as exhaustive review while matching or improving accuracy on three subsets. That supports investigating how information was used, not assuming a provenance edge proves fault. It does not establish an audit budget for your system.

Validate locally before automating attribution

Build a known-answer set from your own failed runs, preserve full traces, and compare a handoff-audit reviewer with a full-trace baseline. Track step and agent agreement separately, plus unsupported certainty, indeterminate cases, and human verification time. OpenAI's guidance supports moving from trace inspection to repeatable datasets and eval runs; it does not independently validate BLADE.

Deciding when multiple agents help addresses whether extra handoffs are worthwhile. Testing false success after a tool failure addresses the separate user-facing report after execution fails.

Bottom line: preserve the source, receiving decision, and task requirement at important handoffs. Test whether the receiver introduced, inherited, or repaired an error, then check the whole run. If the trace cannot support a cause, say so.