How to evaluate an AI agent that changes persistent system state in 2026

If an agent changes a CRM record, issues a refund, schedules work, or calls an infrastructure tool, evaluate the resulting system state—not just its final sentence. Define the required postcondition and prohibited side effects, begin from a known state, inspect the action record and resulting state, and repeat the case before treating the workflow as dependable.

A fluent transcript is not proof of completed work

Salesforce Engineering illustrates the gap with an agent that tells a customer a refund was processed even though the billing record remains open. Its point is narrow but important: when tools change an external system, the conversation and the durable outcome are separate things to verify.

This is a different evaluation problem from asking whether an agent gave a useful answer. A response reviewer can judge a sentence; it cannot by itself establish that a tool was called, that its arguments were right, or that the intended record changed. The broader operating frame is AgentOps and the reliability work around the model.

Write each case as an inspectable state transition

The useful unit is not merely a chat prompt. It is a controlled starting state, an intended durable result, and evidence that lets a reviewer distinguish a completed workflow from a convincing claim.

This comparison is Bonfire AgentOps Hardening guidance, not a fixed vendor schema. The evidence boundary should match the action surface: changing a payment or infrastructure configuration needs a more exact postcondition and side-effect boundary than drafting a report.

Check the outcome and the path separately

Salesforce describes CRMAgentBench as a shared stateful environment that checks actions and final CRM state rather than only the response. Its strict task grading also checks required tools, arguments, execution order, final state, and the absence of forbidden actions or unintended changes. That supports three distinct checks.

1

required postcondition to inspect

1

allowed execution path to constrain

0

prohibited side effects to accept

k

controlled attempts for pass^k evidence

1. Outcome check: did the required postcondition actually exist after the run?
2. Path check: did the agent use an allowed identity, tool, arguments, order, and approval path to reach it?
3. Containment check:did it avoid prohibited actions and state changes outside the case's intended scope?

A pass should not hide a failure in another column. A correct final record reached through an unapproved tool call is not the same result as a correctly executed workflow. Likewise, a reviewer can improve the final response while a system-changing workflow still fails its outcome, path, or containment check; see how to evaluate an agent reviewer.

Repeat the same case before calling it reliable

One successful run shows that a workflow can succeed. It does not show that it will do so consistently. Salesforce describes pass^k as the probability that an agent succeeds on allk independent attempts, in contrast with pass@k, which rewards at least one success. You do not need to adopt a benchmark's metric or threshold verbatim; retain the individual outcomes instead.

• Reached the required postcondition.
• Stopped in a partial state.
• Took a prohibited action or created a collateral change.
• Required a retry, repair, or human handoff.

Inspect those failure modes rather than averaging them into a reassuring score. A partial state that is safely reversible is a different engineering problem from a successful result reached through a forbidden action. They should lead to different fixes and different release decisions.

Make the release decision explicit

NIST's voluntary AI Risk Management Framework organizes risk work around Govern, Map, Measure, and Manage. It says the functions are not a checklist or necessarily an ordered sequence, and describes risk management as continuous across the lifecycle. For an AgentOps release gate, record the decision rather than treating it as a vibe check.

1. State the intended action surface and affected people or systems.
2. Define postconditions, prohibited actions, and required evidence.
3. Run representative cases from a controlled starting state.
4. Decide whether the remaining failure modes fit documented risk tolerance.
5. Block promotion, narrow the action surface, or route to a human when they do not.

Bonfire's stopping-rule guide covers the separate boundary for preserving a champion, enforcing a budget, and escalating when the evidence does not support another repair. NIST's guidance is voluntary and context-dependent; this release rule is an inspectable operating choice, not a certification or universal acceptance threshold.

What to do with a failed case

Keep the failed run, starting state, observed post-state, and action record. Diagnose whether the problem was an incorrect plan, missing information, invalid tool arguments, an authorization failure, an unsafe side effect, or incomplete recovery. Then choose a change that can be tested against the same case.