How to evaluate prompt-injection detectors for AI agent tool outputs

Evaluate a prompt-injection detector where your agent will use it: on representative tool returns, with the deployed wrappers and preprocessing. Freeze a threshold on calibration data, then measure attack detection and benign-task blockage on held-out cases. Keep training-data provenance and the agent's behavior after a detector miss as separate evidence. A generic benchmark score is not a release decision.

In Liu's study of 15 detectors and two task-aware judges, PIGuard detected 95.1% of BIPIA injections but 2.1% of matched AgentDojo injections at a 1% false-positive operating point. Prismor detected 72.2% on AgentDojo and 15.2% on τ-bench. These are benchmark-specific static-attack results, not live defense rates.

Measure what false alarms cost the task

A harmless transaction confirmation can resemble an instruction. If the integration stops a task whenever any return is flagged, per-output false positives can compound into task blockage. In the paper's clean AgentDojo replay, PIGuard flagged 29.5% of 339 benign outputs across 97 tasks at its default threshold; under the study's stop-on-any-flag rule, 59.8% of tasks would be blocked. This is one benchmark example, not a forecast for another workflow.

Build a clean set across the output shapes your agent sees: JSON, search snippets, documents, messages, and confirmations as applicable. Preserve wrappers, field names, source tokens, truncation, and preprocessing. Record both benign-output false-positive rate and otherwise-completable tasks stopped or reviewed under the actual handling policy. The study found serialization and source metadata could shift false-positive behavior.

Test attacks at a frozen operating point

Insert controlled attacks into realistic lower-trust fields without changing the legitimate user request. Pair each attacked call with its clean form and retain the serialized difference. The paper used differential replay because AgentDojo's YAML serialization could re-wrap attack text and defeat simple substring labeling. Its seller-controlled product-name insertion point for τ-bench was a study choice, not a general retail-agent property.

Choose a false-positive budget from your task volume and consequences. Calibrate on one clean partition, freeze the threshold, then report held-out detection and blocked-task rates on different examples. Include sample counts, uncertainty, and per-tool and per-attack-goal slices. The study selected its headline 1% FPR threshold on the same benign set used for evaluation; a held-out threshold test is a stronger proposed release procedure, not one validated by that paper.

Why a benchmark win may not transfer

The study found effectively absent BIPIA-to-AgentDojo detector rank correlation (Kendall's τ = 0.01). PIGuard's released training split contained 1,116 labeled BIPIA inputs. The authors found exact overlap with BIPIA's training-half attack strings, not its test-half strings; the test half shared the input-generation style. Inspect both string overlap and input form. An exact-string audit cannot rule out paraphrases or undisclosed training data.

Include unseen attacks in your own tool-return formats and keep attacker goals separate. The paper found task-irrelevant injections difficult for most evaluated approaches on BIPIA, and adding task context to one small classifier did not reliably improve its low-FPR result. Neither finding proves any detector is universally ineffective.

Keep screening separate from authorization

A flag is a screening signal, not proof the user's task is malicious. A miss does not turn lower-trust text into an instruction. Use instruction-hierarchy tests to measure what happens when malicious text reaches the agent, failure analysis to locate a failed boundary, and the rollout checklist to record the decision. The detector study measured replay screening, not prevented compromise in a live system.

Evidence limits

This unreviewed arXiv v1 preprint used AgentDojo and τ-bench simulations, static attacks, an author-selected τ-bench insertion point, and detailed training-corpus audits for only two detectors. Commercial detector APIs were not evaluated. The results do not identify a universal winner, a universal 1% FPR policy, or live defense effectiveness.