Writing ·
How to evaluate AI agent harness changes without overfitting your evaluation in 2026
Keep the release decision outside the optimization loop: reserve acceptance evidence, meter the search budget, retain the exact candidate and its evaluation record, then run an independent release check.
By Youssef Hemimy · agent evaluation · agent reliability · AgentOps
Treat a harness change as an experiment, not a release by default. Freeze a held-out acceptance set, cap the search budget, preserve the exact candidate and evidence behind each result, and run an independent release check before deployment. A better development score is evidence to investigate—not proof that the agent improved.
- 5
- frontier optimizer models in the cited study
- 4
- downstream tasks in that bounded benchmark
- 111
- scored runs reported by the study
- 1
- reserved acceptance decision per candidate
Keep the release decision outside the search loop
A harness includes the prompts, tools, control flow, context or memory, and orchestration around a model. When an agent changes that harness, the same loop can propose a change, read an evaluation result, and decide what to try next. If release uses only that visible feedback, the loop can select for the evaluator as much as for the job that matters.
HarnessOpt-Bench makes the boundary concrete: its optimizer gets a seed harness, graded development and validation feedback, and a fixed evaluation budget. A nominated candidate is then scored on a test partition inaccessible during search, while a trusted environment meters resources and preserves candidate versions for audit.
Bounded search loop
candidate edits, permitted development feedback, and a fixed budget
Protected acceptance check
fixed release criteria and evidence withheld from search
Evidence ledger
harness, evaluator, cases, traces, budget, and parent version
Release decision
accept, reject, or indeterminate—each with recorded limits
The boundary is the decision: development feedback guides search; reserved evidence challenges the final claim.
Give search and acceptance different jobs
| Control | Search loop | Acceptance check |
|---|---|---|
| Visible evidence | Development cases and bounded validation feedback | Fixed release criteria and reserved cases |
| Budget | Consumes the predeclared evaluation budget | Runs after one candidate is nominated |
| Decision | What to try next | Accept, reject, or mark evidence indeterminate |
A held-out partition is a useful control, not a magic spell. The paper describes its design as hack-resistant rather than hackproof: repeated development and validation feedback can still reward strategies specific to stable evaluator artifacts. The practical question is therefore not whether the latest candidate won a visible score, but what evidence would support a release claim outside the feedback it optimized against.
Meter the optimization budget before reading results
Define the budget before a run in units that fit the system: candidate evaluations, case executions, elapsed time, model tokens, or a combination. The benchmark uses a fixed target-evaluation budget and meters resource use through its execution boundary; its numbers describe that study, not a recommended production threshold.
A practical Bonfire rule is simpler: if a candidate is “almost there,” do not expand the budget silently. Record the expansion as a new experiment. Otherwise the apparent gain has an untracked input: more search.
Keep an evidence ledger for every candidate
If a team cannot reconstruct which harness was evaluated and what it saw, it cannot distinguish a real improvement from a lucky run, changed fixture, or later regression. NIST’s AI RMF Playbook similarly advises documenting the test sets, metrics, methods, and tools behind a result.
- Candidate and parent version.
- Prompt, tool, control-flow, memory, and orchestration configuration.
- Evaluation-set and evaluator versions, including the visibility boundary.
- Budget consumed and stopping reason.
- Per-case outcome, failure class, and trace reference.
- Acceptance result, release decision, and stated limits.
Scale an independent release check to the risk
Independence is a control, not a ceremony. For a low-impact internal workflow, it can mean a separately versioned acceptance suite and a reviewer who did not make the edit. For consequential actions, add domain review, adversarial cases, and a narrow rollout with monitored outcomes. NIST recommends assessing external validity, involving assessors who were not front-line developers where appropriate, and documenting limits on generalization.
The acceptance check should be allowed to return three outcomes:
- Accept: the candidate meets declared criteria on reserved evidence.
- Reject: it regresses, violates a guardrail, or lacks enough evidence.
- Indeterminate: the available evaluation cannot support a release claim.
What to do next
Before using a reviewer’s signal to select a release, evaluate whether the reviewer improves the final answer. Then apply verify–repair stopping rules so the loop does not keep spending budget after evidence stops supporting another revision.
Connect the ledger to an agent observability dashboard, and make it part of the operational AgentOps discipline. A score should be the start of an explanation, not the end of the review.
Sources
FAQ
Is a held-out test split enough to prove a harness change is safe?
No. It is one useful boundary, not proof of safety. The cited preprint notes that repeated development and validation feedback can still reward behavior tailored to stable evaluator artifacts.
Should the agent that changes the harness see no evaluation feedback?
No. Bounded development feedback is useful for iteration. Reserve a separate acceptance check for the release decision so the same evidence is not both the search target and the sole proof of improvement.
What should an evaluation ledger contain?
Retain the candidate and parent version, evaluator and evaluation-set versions, allowed budget, per-case outcomes, trace references, and the acceptance decision. NIST separately recommends documenting test sets, metrics, methods, and tools used in evaluation.
Does the research establish a standard budget or the best harness optimizer?
No. The study reports its own bounded protocol—five optimizer models, four tasks, and 111 scored runs—and limits generalization to its current benchmark conditions.
Building something that has to hold up?
We do this work for teams — agent reliability hardening, custom MCP servers, and full-stack AI systems built to survive production.