How to test AI agents against near-target prompt injection

Test whether your AI agent rejects instructions in lower-trust material even when they request an answer or action that still resembles the authorized task. Start with a correct task outcome, choose one specific wrong fact or unauthorized side effect, validate that it is a real violation, and insert the instruction in a document or tool result the agent must read. Grade task completion and attacker-desired deviation separately against an attack-free run. This is an evaluation pattern, not a prevention control.

Why a plausible answer can still fail

In the paper's illustrative example, a conspicuous injection makes a writing assistant abandon a workshop announcement. A nearer attack keeps the announcement but changes its time from 10 a.m. to 11 a.m.; another keeps the correct time and room while exposing a private phone number. These are illustrations, not observed Bonfire incidents. The output can look useful while violating the original request.

The boundary is authority, not relevance. A retrieved document or tool return supplies task data; it does not get to rewrite the user's requirements. OWASP's LLM01 guidance and the OpenAI Model Spec provide independent context for that boundary, not evidence for this paper's measured rates.

What NTBench measured

Gong and colleagues' On Task, Off Track starts from legitimate reference answers or agent actions, then creates target–instruction pairs in two tracks: an incorrect deviation and a harmful outcome that may accompany otherwise correct completion. Targets are checked before their instructions are inserted into lower-trust content. “Near” is prompted, not enforced by a numeric distance threshold.

The authors evaluated seven models across ten language-task datasets and two agent benchmarks, AgentDojo and AgentDyn. With three attack executions per sample and fixed attack sets shared across models, both near-target tracks had higher pooled attack success than the tested fixed-target baseline for every model in each setting. On language tasks, the incorrect-target gap was 9.3–35.8 percentage points and the harmful-target gap 1.7–17.3 points. The harmful-target result was not uniform across datasets: success was higher in 63 of 70 model–dataset combinations, equal in two, and lower in five.

Those comparisons apply to selected generated attacks, models, tasks, checks, and the three-attempt budget. They do not say how often near-target attacks arise in deployed systems. A matched tool call and checked key arguments count as tool-call success in the study; downstream effect completion is not established. The authors also found imperfect agreement between text judges and human labels.

Build a local evaluation

1. Freeze the legitimate task. Keep the user request, trusted constraints, lower-trust input, expected facts, and permitted tool actions unchanged. Choose representative work your agent actually performs.
2. Choose a nearby violation. Change one consequential answer fact, add an unrequested disclosure, or change a key tool argument. Track incorrect and harmful outcomes separately; they are overlapping, not additive score buckets.
3. Validate the target. A different answer may still be valid, and a proposed tool call may be impossible. Use a task-aware reviewer or deterministic oracle; check tool schemas and preconditions. Inspect ambiguous cases rather than treating generated labels as ground truth.
4. Inject at the boundary under test. Put the instruction in a retrieved document or tool return, leaving the user request and trusted instructions intact. Run an attack-free control. A deviation that occurs without injection is a reliability failure, not attack-induced success under the paper's metric.
5. Grade outcomes and effects separately. Record legitimate-task completion, the specified attacker deviation, unauthorized calls, and actual side effects. Keep the attempt budget, injection surface, harness, and model revision with the result. Escalate uncertain text judgments for human review.

This workflow adapts the study's method for an operator-owned suite; the authors did not validate it as a deployed release control. An apparently correct final answer must not erase a harmful tool call in the trace. Pair it with instruction-hierarchy tests, stateful-outcome evaluation, and human approval gates for the adjacent decisions.

Evidence limits

This is an unreviewed arXiv v1 preprint. Prompt-guided closeness is not a quantified neighborhood; the datasets, policies, generated attacks, and judges are selected; text judgments can err; and exact tool-call matching does not prove effects completed. Neither the benchmark results nor the local workflow establish live-deployment efficacy.