How to evaluate an enterprise AI agent on facts spread across systems

Test whether the agent reconciles the right records, not merely whether it retrieves a plausible answer. Build a read-only case with an accessible clue, a tempting but wrong single-system answer, and an exact key computed independently of the agent. Grade the complete answer and inspect the records it used before treating a pass as evidence for a release decision.

This is a test-design pattern from the company-authored Era by Eon preprint, not evidence that an agent is ready to act on live contracts, customers, or business systems.

A question that exposes the single-system shortcut

In one fictional Era case, a contract record says a customer's service credit is zero. A recorded sales call promises five percentage points for each month with an urgent support ticket, capped at 15. The agent must find the call, inspect the right tickets, and apply the rule to the contract term. The benchmark's exact key is a 15% credit starting in December 2023 and covering ten months. Reading only the contract yields a plausible but wrong answer.

The evaluation question is which evidence governs the constructed answer and what computation follows. A recorded promise would not automatically create a legally valid credit in a real company; organizational policy and legal review would decide that.

Four controls before a case enters the suite

Era requires that the decisive evidence be served to the agent, the answer be unique, the obvious answer be wrong, and the output be exact. Its generator computes the key without a language model, re-derives it from the served records, and drops invalid questions. Turn those conditions into a case review:

These controls make a test interpretable; they do not make synthetic records representative of production by themselves.

Separate finding the clue from computing the answer

Era's eight hidden-fact templates include four call-remark cases, three record-selection cases, and one older-report-method case. The simulated read-only systems include CRM, help desk, call transcripts, issue tracking, and files. Retain an evidence path beside each key: subject identifiers, decisive records and sources, near-matches or conflicts, the rule applied, and computed fields.

Review a failed run in stages. Did the agent select the wrong account, miss the call, use a stale proposal, join the wrong ticket, or apply the right rule to the wrong time window? That diagnosis is more actionable than one aggregate pass rate. Inspecting this evidence path is Bonfire guidance, not a claim that Era formally grades every intermediate step.

Interpret the result within its tested boundary

The paper tested six models paired with two agent programs on eight hidden-fact questions from one generated large fintech company. Each pair made three attempts per question; the strongest answered 18 of 24 attempts correctly. On an earlier set of 27 computable questions, stronger code-enabled agents answered 22–25 correctly, motivating the authors to add cases where an unstated fact had to be inferred.

These counts show errors in this setup, not a general model ranking or expected performance on live data. The applications are simulated, the hidden-fact suite has eight templates, and the agents are read-only. An exact key inside a fixture does not eliminate real-world ambiguity, permission limits, stale data, or policy disputes.

Use the score in a contextual release decision

The NIST AI RMF Core and Playbook organize voluntary, context-tailored risk work around Govern, Map, Measure, and Manage. They do not prescribe Era's benchmark or a universal pass threshold. Document the business questions, authorized data scope, harmful wrong answers, and local evidence required before promotion.

1. Version the fixture, answer key, and authorized read tools.
2. Audit the four case controls and run repeated trials.
3. Retain final answers alongside evidence-path and conflict-resolution failures.
4. Compare those failure modes with representative workflow data and permissions.

If the live agent can write, add persistent-state and prohibited-side-effect checks. Bonfire's stateful-agent evaluation guide covers that separate gate. For monitoring and recovery after release, see what AgentOps is.