How to evaluate browser agents when the target is hidden

Test whether a browser agent can make its target visible before testing whether it clicks the right place. Add tasks with off-screen, hover-only, collapsed, occluded, delayed, and moving targets. Record the reveal action and resulting interface state separately from final-click success. A single end-to-end pass rate hides which boundary failed.

What the IGG benchmark isolates

Kim, Son, and Oh's Interactive GUI Grounding (IGG) paper tests 601 tasks in 16 deterministic, application-like web environments. The agent receives screenshots, a goal-only instruction, and its action history. It can scroll, move the pointer, drag, wait, and click; the evaluator checks whether a click hits a stable target identifier within an eight-action budget. The DOM, accessibility tree, target ID, and reference trace are hidden from the tested model.

The seven subtypes cover off-screen targets; hover-based disambiguation; collapsed or inactive regions; removable popups; delayed activation; changing targets such as carousels; and movable occlusions. This is narrower than full browser navigation: IGG isolates the interaction needed to expose a target and one evaluated click, without scoring a long workflow or its side effects.

On the complete benchmark, GPT-5.5 reached 79.4% grounding accuracy and the strongest evaluated open-source VLM reached 31.6%. A separate human solvability study achieved 94.6% on 315 trials across a 21-task subset; it used a different attempt-counting rule, so these numbers are not a like-for-like human–model leaderboard. For Qwen3-VL, naming the required reveal interaction changed accuracy from 30.8% to 33.9%, while performing the reference reveal before the model's click raised it to 81.9%. The intervention points to reveal execution in this setup; it does not mean all deployed-agent failures share that cause.

Turn a hidden-target failure into a useful test

1. Start from an unexposed target. Save the initial viewport, browser state, instruction, intended target identity, and permitted actions. Do not leak the reveal procedure in the instruction.
2. Instrument the reveal. After each action, capture the screenshot and whether the intended element became visible and actionable. Distinguish a correct action category from a correct location, amount, timing, or state transition.
3. Score the final click independently. Record target identity, wrong clicks, action count, and elapsed time. A revealed target followed by a wrong click is a grounding failure; a correct-looking click before the target is actionable is not a reveal success.
4. Replay on your actual interface. Vary viewport, delays, overlays, and dynamic states. Keep the same model, browser harness, action budget, and evaluator when comparing revisions. Review consequential clicks and their side effects separately.

This test design is an operator adaptation, not an intervention validated by the paper on deployed sites. Playwright's actionability documentation independently distinguishes visibility, stability, event reception, and enabled state when automating clicks; its input guidance covers hover, scrolling, and drag actions. Those automation checks can help construct or inspect test states, but passing a Playwright locator check does not establish that an agent selected the right reveal action. For the adjacent safety decision, see checking browser clicks against user intent. For release evidence, see evaluating harness changes without overfitting.

Evidence limits

IGG is an arXiv v1 preprint using deterministic mockups and screenshot-only observations with five actions and an eight-action model budget. Its authors report a preliminary paired mock–real comparison on just 16 tasks, one per environment, and explicitly say the mockups do not replace live-interface evaluation. The paper does not test DOM- or accessibility-tree-driven agents, arbitrary websites, long workflows, or production safety outcomes.