How to verify a computer-use agent actually completed a GUI task

A click, a success message, and the agent's “done” are not the same as a completed task. Name the mandatory result, then inspect post-action evidence that could prove or refute each part. A visible screen can establish what was displayed; a permitted read of a file or application record can establish what persisted. If neither is available, record unknown rather than promoting the agent's claim to success.

Start with the result, not the final message

Suppose the task is to create a note with a particular filename and text. The hard requirements are that the note exists, is in the intended location, and contains the requested text. Seeing text in an editor does not prove it was saved. A read-only file or application-store check can address persistence when that state is accessible. VeriCUA's worked example separates these checks and treats the submitted answer as a claim to investigate.

Inspect available app context before writing the rubric, so an assumed path or UI element does not silently become a success condition. Mark necessary outcomes as hard requirements and supplementary diagnostics as soft ones. A soft observation cannot compensate for an unproven hard outcome.

Give each requirement a source-linked check

Link each finding to an executed read and check its timing and persistence. In VeriCUA's arbitration, success requires support for every hard requirement; a refuted hard requirement means failure; insufficient evidence remains unresolved. The paper's offline configuration may fall back to an initial vision-model judgment for unresolved recordings. That fallback is not newly observed proof.

For web interfaces, an auto-retrying assertion that a confirmation element says “Submitted” is a useful surface check, as Playwright's assertion documentation describes. If the requirement is a saved record, add an authorized readback. Keep verifier reads separate from mutation tools.

What the experiment actually shows

On the paper's online AndroidWorld evaluation, VeriCUA reported 90.5% agreement with official labels and a 7.7% false-positive rate, versus 78.4% agreement and 15.4% false positives for its single-pass vision-model judge. But it made a decided judgment on only 82% of episodes, while the baseline always returned one. The authors count abstentions as failures in the reported classification metrics. On OSWorld and ScienceBoard, a generated-verifier baseline had higher recall even as VeriCUA reported higher accuracy and fewer false positives.

For recorded trajectories, the largest reported accuracy gap was on OSReward-Hard: 70.4% for VeriCUA versus 65.7% for the strongest compared judge. These figures measure agreement with study reference labels, not independent real-world correctness. A recording cannot reveal state that was never retained; the paper also notes possible reference-label errors.

A small adoption gate

1. Choose checkable outcomes. Name required postconditions and the evidence source for each. Approval to act is separate from proof of completion.
2. Preserve observations. Retain time-stamped action and UI evidence and, where authorized, before/after or read-only application-state views. See the broader stateful-outcome evaluation guide.
3. Return three outcomes. Report confirmed, refuted, or unknown for each hard requirement. Escalate unknowns; investigate refuted claims with a root-cause workflow.
4. Measure the tradeoff locally. Compare false approvals, false rejections, undecided rate, and cost on your task mix. VeriCUA's Appendix E describes tens of tool and model calls per pass, roughly an order of magnitude more than a single-call judge.