How to set release acceptance thresholds for an AI agent in 2026

Set the release rule before you look at the score. Version representative cases, declare success and safety criteria, set limits for each criterion, and retain the evidence behind an accept, reject, or escalate decision. A convincing demo is not a release gate.

A threshold is a decision contract, not a leaderboard score

The useful question is not “what score should our agent get?” It is: what evidence would make this specific workflow useful enough and controlled enough to promote, and what evidence must block it?

This table is Bonfire AgentOps Hardening guidance. It applies NIST's voluntary, context-dependent advice to choose metrics for the evaluation's purpose, define acceptable performance limits and corrective actions, and document unmeasured risks.

Give each criterion a rule shape

Microsoft Foundry documents rubric evaluators with weighted dimensions, along with agent, quality, safety, and custom evaluators. That supports a practical distinction: quality measures can have a declared range, while a confirmed non-negotiable control failure should block promotion.

The template does not claim every agent needs every measure. For a workflow that changes an external system, include the durable postcondition and prohibited side effects—not merely the final message. See how to evaluate an AI agent that changes persistent system state.

Make the task set part of the release claim

Microsoft's documented workflow uses a test dataset of queries, runs each query through the agent, and applies selected evaluators to the result. Before a release run, make the task set inspectable:

1. Record the supported job, allowed tools, and assumed starting state.
2. Include ordinary, difficult, and explicit refusal or approval cases.
3. Version every case, fixture, and reference artifact.
4. Record material blind spots instead of omitting them from the chart.

NIST recommends documenting the test sets, metrics, and tools used in testing, evaluation, validation, and verification. A task set is not proof that every production condition has been covered.

Use an explicit promotion rule

A threshold should produce a decision without re-litigating the rule after results arrive. The exact numbers belong to the team and risk boundary; the decision shapes do not need to be vague.

Evaluate whether an AI agent reviewer improves the final answer before using that signal in a release gate. Keep a final acceptance boundary outside any optimizing harness loop; this guide to harness-change evaluation explains why.

Retain failures, not only the final percentage

Microsoft documents aggregate pass/fail counts and results per evaluator for an evaluation run. The aggregate is the summary, not the whole evidence. Retain the exact agent, prompt, tool, model, policy, and evaluator versions; task-set and fixture versions; per-case outcomes and available traces; thresholds in force; excluded risks; and the accept, reject, or escalate decision.

If the evidence no longer supports another revision, stop rather than expanding the effort silently. Use verify–repair stopping rules for that separate decision.

The caveat: a threshold does not prove production safety

A passing threshold means the candidate met the stated rule on the stated evidence. It does not establish that the task set was complete, that an evaluator was correct, or that the agent will behave safely in every production condition. Microsoft's documentation is product-specific; NIST's guidance is voluntary and adaptable. Keep access controls, approvals, containment, rollback, and production monitoring where the workflow requires them.