How to build a prompt-injection red-team corpus for AI agents

Build a prompt-injection red-team corpus as versioned executable security cases—not a folder of clever strings. Bind each attack to its benign task, untrusted context, malicious objective, protected action, expected safe result, observed trace, and outcome. Keep attack discovery separate from held-out release evidence, then replay the suite after every material agent change.

That makes prompt-injection testing a release control. A copied payload without its task, authority boundary, and expected result is trivia. An executable case can show whether one versioned system preserved one protected decision under one recorded threat condition.

A useful corpus case is a decision record

Start with the action you are protecting. Malicious text matters only in relation to the legitimate task, the untrusted surface carrying it, and the side effect it tries to cause.

This schema is Bonfire operating guidance, not a standard copied from either source. NIST frames a hijacking scenario around a legitimate user task and a malicious injection task. The PIMiner preprint describes human-readable strategy files that preserve target and task scope, examples, fingerprints, and explicit failure conditions.

Separate discovery, curation, and release evidence

Discovery finds candidate attacks

Run authorized attack generation in a sandbox with synthetic identities, inert secrets, bounded tools, and no path to production side effects. Discovery may iterate, combine known strategies, and learn from failed attempts. Its output is a candidate case—not release evidence and not permission to test someone else's system.

PIMiner is relevant because it does not restart every sample from zero. In its benchmark setting, a router selects from a long-term strategy library, the attacker retains experience from earlier samples in a dataset and the current sample's history, and a digester updates strategies from successes and misses.

Curation turns discoveries into stable cases

A security reviewer should decide whether a discovery adds a distinct mechanism, risk, or coverage boundary. Normalize it into the corpus schema, remove irrelevant payload variation, retain its provenance, and record where it has and has not worked. Deduplicate by protected decision and attack mechanism, not only by string similarity.

Do not erase misses. A failed attack can show that a strategy depends on a particular tool, task shape, context slot, or target behavior. That is evidence about scope, not clutter.

A held-out suite makes the release decision

Freeze an acceptance set before tuning the candidate system. Do not let the same cases guide a defense change and then become the only proof that the change worked. NIST reports developing adaptive attacks on a subset of Workspace tasks and evaluating them on held-out tasks. The exact split is not a universal recipe; the important boundary is that discovery did not consume all acceptance evidence.

Use at least three corpus buckets:

• Development cases for finding and fixing failures.
• Held-out acceptance cases for the promotion decision.
• Incident regressions for every confirmed real or pre-production failure, preserved without quietly rewriting its expected result.

Preserve transfer evidence without claiming universal attacks

PIMiner's experiments use controlled public samples from IPIArena and AgentDojo, with separate training and test splits. The paper reports benchmark-bounded transfer of its strategy library to target models outside its training sequence, and its ablations find that the strategy library and within-dataset memory both contributed in the tested IPIArena setting.

The practical lesson is narrower than “this attack transfers.” Record the dimensions across which transfer was actually observed:

• agent and model version;
• tool and permission surface;
• benign task family and untrusted-content surface;
• malicious objective and protected action;
• action policy and confirmation boundary; and
• number of attempts and query budget.

When a strategy succeeds on another configuration, add that evidence. When it fails, preserve that too. Do not turn “worked on one benchmark target” into a claim about a private workflow, a future model, or every agent using the same framework.

Measure risk by task and repeated opportunity

An overall attack-success rate can hide the case that matters most. Record:

1. Benign task completion: did the agent still complete the legitimate work?
2. Malicious objective completion: did the prohibited outcome occur?
3. Protected-action decision: was the request denied, constrained, approved, or escalated according to policy?
4. Impact class: what would the attempted action have changed or exposed?
5. Attempt profile: did the case fail once, intermittently, or whenever an attacker could retry?

NIST recommends inspecting task-specific performance in addition to aggregate results because injection tasks differ in success frequency and impact. It also recommends multiple attempts where attackers can realistically retry because model behavior can vary between runs.

Set the repeat count and release threshold from your threat model before the run. These sources provide no universal number. A read-only assistant and an agent allowed to send messages, run code, or move money should not inherit the same veto policy because a dashboard prefers one tidy average.

Promote cases with a controlled workflow

1. Discover in a sandbox.
2. Review authorization and provenance.
3. Classify the protected action and risk.
4. Normalize the finding into an executable case.
5. Deduplicate by mechanism and decision boundary.
6. Validate the expected outcome.
7. Assign development or held-out status.
8. Replay on a versioned system and retain scope, outcomes, and failures.

Require human review before a case enters the shared corpus when it contains a novel mechanism, realistic sensitive data, or a consequential action. Store attack material with access controls proportionate to its risk. Use synthetic identities and inert credentials. Verify final state instead of trusting the agent's narration.

Replay the relevant corpus slice after changes to:

• the model or model alias;
• system or developer instructions;
• tool schemas, permissions, or confirmation rules;
• retrieval, memory, or context assembly;
• content screening or policy services; or
• retries, orchestration, or recovery behavior.

Connect the corpus to enforceable controls

Use instruction-hierarchy tests to encode the expected action, refusal, or escalation. Run discovery and replay inside an isolated agent-evaluation environment. Set per-risk vetoes with AI agent release acceptance thresholds, then enforce the protected action and allowed parameters through production agent tool contracts.

What this corpus can and cannot establish

A passing corpus says that a versioned system met explicit expectations on the cases and attempts you ran. It does not prove the absence of prompt injection, cover unknown delivery surfaces, or convert a benchmark result into a production risk estimate.

The PIMiner evidence is one unreviewed preprint. Its experiments use public, sandboxed benchmarks rather than real systems, users, or private data, and the paper primarily uses Claude Code models for its red-team backbone. NIST's work is evaluation guidance, not a certification scheme.

The useful claim is deliberately smaller: this system, at this version, preserved these protected decisions under these recorded attacks. “Prompt-injection safe” is marketing fog with a clipboard.