How to stress-test a long-running multi-agent system for adversarial failures

Test a long-running agent system after it has representative memory, tools, work artifacts, and shared state—not only from a blank session. Introduce one bounded adversarial event through a normal input surface, then check separately whether the system detected it, prevented consequential propagation, recovered the intended state, and removed the contamination it created.

That is an engineering test design, not a claim that a single harness can certify a system as safe. The point is to make persistence visible before promotion. For a narrower action-boundary test, start with testing instruction hierarchy in AI agents.

Why a fresh prompt test leaves a gap

In the Emergence World preprint, researchers ran eight simulated ten-agent worlds with persistent memory, tools, and shared institutions, then introduced controlled stress events only after the worlds had accumulated operating history. The events covered indirect prompt injection, misinformation, and exposure of private agent memories through ordinary interaction surfaces. Source: Akkil et al.

The reported setting is deliberately unlike a short, isolated prompt check: prior work, saved information, and peer interactions could influence what happened after the event. The authors report that, in their simulated worlds, recognizing a threat did not necessarily mean the threat was restrained, contained, or followed by recovery. Source: Akkil et al.

Design the test around an event and its consequences

OWASP describes its Agentic AI – Threats and Mitigations guide as a threat-model-based reference for emerging agentic threats and mitigations. Source: OWASP Use that framing before choosing test content: name the interaction surface, what the event is trying to influence, and what consequence would matter.

For each event, write a compact test card before running it:

This is Bonfire AgentOps Hardening guidance. The exact surfaces and constraints must come from your system's threat model; do not copy the preprint's stimuli as though they cover every workflow.

Run a stateful event, then use four independent checks

The paper's event rubrics distinguish recognition, restraint, containment, coordination, and durable response. Source: Akkil et al. A practical release check can simplify those into four independently recorded outcomes:

1. Detection: Did the relevant control or operator receive a usable signal? Record the signal, timestamp, and evidence it relied on.
2. Containment: Did the system prevent the event from reaching a consequential tool, memory store, shared artifact, or peer workflow? Inspect the paths you listed in the test card.
3. Recovery: If anything crossed the boundary, was the required safe postcondition restored without introducing a second harmful change?
4. Cleanup: Were contaminated memories, queued jobs, cached artifacts, or access grants removed, quarantined, or marked so that ordinary work will not reuse them?

Keep these columns separate. A warning is not containment. A blocked tool call does not prove that an earlier retrieval was not stored. A restored record does not prove that a queued follow-up will not replay the event tomorrow. That separation is the value of the test.

For durable-state inspection, use the method in evaluating an AI agent that changes persistent system state: define the required postcondition and the changes that must remain absent, then preserve evidence that shows both.

Use a warm-up that is representative, not theatrical

The preprint injects events after state accumulates; it does not supply a universal amount of history that every agent needs. Source: Akkil et al. Choose the warm-up from the dependencies that make your system different from a fresh session:

• create ordinary task artifacts the agent would later read or modify;
• populate only the memory and retrieval records that the intended workflow can actually access;
• exercise the normal tool and handoff paths that might carry an event forward; and
• use identities and permissions that match the release scope, including the boundaries that should deny the action.

Then reset the fixture to the captured starting state for each repeat. If an event cannot be replayed from a known state, its result is harder to interpret and harder to compare after a control changes.

Decide promotion from the escape path, not a single score

Before promotion, require the test record to answer four questions:

1. Which state was present before the event?
2. What did the system detect, block, change, and preserve afterward?
3. Did the event reach any consequential path that the test card declared protected?
4. If it did, was the repair and cleanup verified before normal work resumed?

There is no universal pass count or safety threshold in the sources. A defensible decision instead names the release scope, the events that matter for it, the observed failure modes, and the control or operating boundary that addresses each one. If containment or cleanup is unproven for a consequential path, narrow the release scope or keep a human approval boundary while the evidence is incomplete.

When a case fails, do not jump straight to another run and call the newest outcome progress. Preserve the starting snapshot, injection, trace, post-state, and cleanup result; then use an evidence-backed root-cause analysis to find the earliest failure that materially changed the outcome.

Caveats

• Emergence World is an arXiv v1 preprint, not peer-reviewed production evidence. Its findings are limited to eight simulated worlds and three controlled event types.
• The study motivates a stateful test pattern; it does not establish that every long-running system will exhibit the same failures, or that its test setup represents your environment.
• OWASP's threat-model reference supports making the event surface and consequences explicit. It does not prescribe this scorecard or certify a system as secure.
• A successful stress test complements access control, monitoring, rollback, incident response, and human review; it does not replace them.