How to test multilingual AI-agent guards before deployment

Before deploying an AI-agent tool-call guard to multilingual users, replay matched, labelled steps that differ only in the language of the user's instruction. Keep the proposed action, trace, policy, label, and guard settings fixed. Measure not just verdict flips, but newly blocked legitimate actions and missed unsafe actions by language. Compare against English rewording and a placebo/noise baseline before expanding the guard's use.

What should stay fixed in the test?

Begin with a labelled step: user request, prior agent trace, candidate tool call, and allow/block label. Translate only the user request while preserving names, paths, quotes, amounts, dates, permission scope, and other task-defining details. Review the translation for equivalent intent and retain a versioned record of each variant. Keep the proposed tool call, history, environment, label, guard prompt, parser, threshold, and decoding settings unchanged within a pair.

Demirkiran and Ernstberger used this fixed-step design on TS-Bench material derived from AgentDojo. They translated instructions into German, French, Turkish, Swahili, and Amharic, then compared guard decisions with English originals. Their English paraphrase and whitespace-placebo controls help distinguish language effects from ordinary wording or run variability. Repeat nondeterministic guard calls and compare each translated result with its matching original step, not an aggregate of different tasks.

This test deliberately isolates one guard input. If the deployed agent's reasoning and tool outputs also change language, add a separate end-to-end multilingual workflow evaluation; the fixed-step result cannot stand in for it.

Which outcomes determine release readiness?

Report paired verdict flips above the placebo floor, then split the errors. Count safe calls newly blocked (false-positive change) and labelled unsafe calls no longer blocked (unsafe-step recall change) for each language and guard configuration. A flip alone is ambiguous: it may correct an English mistake. Preserve paired examples, counts, and uncertainty instead of averaging away a weak language. Set the acceptance rule for the tool action's risk before looking at the results.

On a separate representative workflow set, measure ordinary-task completion. That utility check is an operational recommendation, not an outcome established by the fixed-step paper. In its tested roster, several 12–30B open guards tended to over-block benign steps after translation, while statistically significant unsafe-step recall losses appeared in guards below 10B. Some frontier guards barely moved. These are study-specific tendencies, not a size threshold or a guarantee for a deployed guard.

How should a failed language check be interpreted?

Investigate the paired cases before promoting the guard. The authors did not test translation-first guarding, language-specific thresholds, or other mitigation, so the result is not a recipe for automatically translating every request to English. Their post-hoc account suggests that translated requests may conflict with English restatements in an agent trace, but this is a hypothesis for the benchmark, not a proven universal cause. A stable verdict can also be misleading: one strong guard produced the same decisions when the request was withheld.

The 1,220 steps arise from only 58 underlying instructions. The authors caution that their uncertainty intervals may be too narrow because translations were assigned per instruction but bootstrap uncertainty was clustered by trajectory. Lower-resource translations had additional validation limits, and not every guard ran in every language. Offline fixed-step replay did not execute attacks. None of these results establishes real-world attack frequency.

Use the same evidence discipline as instruction-hierarchy testing, and treat language as a representation variable alongside tool-name sensitivity. Where a language materially shifts errors, investigate the cause and verify ordinary-task utility before widening the release.