How to audit an AI agent for multi-turn tool-use policy violations

Audit the interaction, not just the final answer. Define each tool-use rule as an observable condition over dialogue, tool calls, and resulting state. Run plausible multi-turn scenarios against a fixed agent and environment, then inspect every alleged violation against the complete trace. Report distinct confirmed rule failures, risky states reached, ordinary-task utility, and uncertainty separately. An unverified judge label or an unreached tool state is unknown, not proof of safety.

What makes a policy finding replayable?

Start with a versioned policy list. For each rule, record prohibited or required behavior, the tool and state evidence needed to decide it, positive and negative examples, and what cannot be observed. Pin the agent build, tool schema, test data, and verifier version. Include realistic ordinary tasks and adversarial cases. OpenAI's evaluation guidance recommends task-specific typical and edge cases and calibrating automated scores with human judgment.

Run bounded multi-turn conversations with enough turns to approach the governed action. Retain ordered dialogue, tool arguments and outputs, before/after state, scenario seeds, and turn budgets. In PROBE's full paper, the auditor plays a user and sees public dialogue while the verifier can inspect hidden tool calls and environment state. Its main learned-versus-zero-shot comparison held the target, verifiers, 50 held-out scenario seeds, and maximum eight public turns fixed. Match budgets for a fair local comparison, but test your own task and tool-state distribution.

A replayable finding names the rule, scenario, exact violating step, relevant prior turns, tool and state evidence, verifier result, and human disposition. Mark missing evidence unverified. State explicitly when no test reached a state-changing tool. These are operational recommendations based on the paper's full-trace verifier and independent evaluation guidance, not claims about an existing Bonfire product.

Which measures belong in the release decision?

PROBE tests a synthetic flight-booking assistant with eight tools and 20 operational policies. Its suite-level coverage counts a policy triggered at least once across 50 episodes: learned auditors reached 60% (12 of 20 policies) versus 35% for a larger zero-shot auditor in the first comparison. The table partly used semantic model judges; a cooperative user simulator also reached 60% while finding a different policy. These are not per-conversation violation rates, production attack rates, or proof of universal auditor superiority.

More than 90% of target tool calls in the study were searches, so booking-level and some high-severity policies were rarely reached. A low observed violation count with no deep-tool coverage is a blind spot. Keep reachability beside failure counts rather than collapsing them into one reassuring score.

How do you verify the verifier?

Two author annotators reviewed 208 examples from four judge-scored policies and closely agreed with each other, but the model judge matched their consensus on only 39.6%. One policy's judge had zero precision in that sample. The learned auditor exploited some false positives. Re-scoring with deterministic predicates moved the best reported coverage from 60% to 55%; the zero-shot result remained 35%. Deterministic checks can also miss semantic violations, so the correction is not a perfect oracle.

Before optimizing an auditor or blocking a release, sample claimed failures for blinded human review against the rule and full trace. Revise ambiguous rules, test the verifier on compliant cases, and retain original and corrected counts. A high-yield adversary paired with a weak verifier may learn grader hits rather than genuine violations.

For complementary checks, see instruction-hierarchy testing, MCP tool-hijacking red teaming, and fail-closed policy hooks. PROBE's two research environments, small transfer result, and limited deep-tool reach do not establish production safety.