Writing ·
Check coding-agent tool calls against repository state before execution
Check repository-state prerequisites before coding-agent tool calls; block confirmed mismatches while measuring false blocks and task outcomes.
By Youssef Hemimy · coding agents · agent reliability · tool contracts
Before a coding agent executes a tool call, check the assumptions that can be verified against its current workspace. Extract concrete prerequisites from the proposed action—working directory, file path, importable package, or exported symbol—and probe those read-only. Return the exact mismatch to the agent before execution. Keep this validity check separate from authorization and from whether the action actually solves the task.
Put a narrow state check at the action boundary
The StateMirror paper intercepts a proposed coding-agent action, parses it into typed predicates, and probes the current repository and harness state without running the action. Its motivating examples include a nonexistent path, a module layout guessed backward, and a symbol whose capitalization differs from the actual export. The check is specific: which assumption failed, against which workspace snapshot?
1 · Proposed call
Capture action, arguments, and current workspace.
2 · State predicate
Check path, package, or symbol read-only.
3 · Typed result
Mismatch → diagnose; unknown → policy; match → continue.
4 · Independent gates
Authorize the action, then verify its outcome.
A prerequisite match does not grant permission or prove task success.
| Proposed assumption | Read-only check before execution |
|---|---|
| File or directory exists at this working directory | Resolve the path against the current workspace and test existence, without changing it. |
| Package or module is importable | Inspect the active environment and package resolution, not a different developer checkout. |
| Named symbol is available | Inspect the module or export surface and preserve case and scope. |
A harness can intercept before execution: the OpenAI Agents SDK tool-guardrail documentation describes a pre-invocation seam for guarded function tools, but it does not supply StateMirror's probes or cover every shell path. These checks have ordinary language/runtime equivalents: Python's Path.exists tests path existence and importlib.util.find_spec looks for a module specification. Neither by itself supplies StateMirror's action parser, resolves every dynamic import, or proves a proposed edit is good. Probe the same container, worktree, dependency set, and working directory that the tool will use.
Return a typed mismatch, not a blanket veto
Capture the proposed call, extracted predicate, observed state, and decision in one trace. Distinguish false from unknown: generated paths, dynamic imports, conditional state changes, and concurrent edits can defeat a static predicate. Recheck immediately before the tool runs if the workspace can change between probe and execution. Do not let an absent path check silently override an action that is meant to create that path.
For a proven mismatch, return a short diagnostic such as “module textual.src is absent under this environment; inspect src/textual.” Bound the agent's revision attempts. A successful retry still needs normal tool authorization, execution checks, and task-outcome verification. For adjacent boundaries, see production tool contracts, tool-interface drift repair, and combined edit acceptance.
Read the evidence as a canary, not a threshold
The authors' preliminary study reviewed 4,398 SWE-agent trajectories and identified 9,595 hallucinated tool calls among 120,801 calls. In a separate human-labeled 2,508-action set drawn primarily from failed trajectories, StateMirror reports 0.940 precision, 0.722 recall, and 0.817 F1, with 64 false positives and 390 false negatives. Median detection latency was 1.50 ms, but p95 was 306.66 ms and the slowest probe took 11.73 seconds in the authors' setup. These are detector results, not a production base rate or capacity plan.
In a separate resolved-run stratum, recall was lower at 0.532, including misses tied to dynamic runtime state. The paper's 79-task intervention deliberately selected runs with an actionable detected mismatch: blocking reduced average time 15.8% and tokens 11.8%, while both modes resolved 48 of 79 tasks. Of 78 first replacements, 73 cleared warned predicates but only 66 executed successfully. This is a triggered-subset efficiency result, not proof of general task improvement. The paper notes lower recall on resolved trajectories where errors depend on dynamic runtime state. Its broader evaluation is concentrated in Python bug-fixing tasks and the specified agent/model setup. Extractors and probers would need adaptation for other languages and runtimes. A read-only check also does not catch valid-but-wrong actions or unauthorized side effects.
Canary the gate against your own actions
- Replay representative accepted and failed calls from the same harness and workspace snapshot. Label true mismatches, valid creates, dynamic unknowns, and consequential actions.
- Start with high-confidence path and package predicates. Log extracted checks, false blocks, missed invalid calls, and stale-state races.
- Compare a shadow checker with the current reactive workflow on task resolution, recovery calls, latency, token cost, and review burden.
- Only then enable a narrow pre-execution block for proven mismatches; preserve authorization checks and post-execution outcome validation.
Sources
FAQ
What can a pre-execution check catch?
Concrete mismatches between a proposed coding-agent action and the current workspace, such as a missing path, package, or exported symbol. It cannot certify that an executable action is correct, authorized, or safe.
Should an unknown precondition block every tool call?
No. Report unknown separately from a proven mismatch. Route consequential unknowns to a stricter policy or review, and measure false blocks against accepted task outcomes before tightening the gate.
Does StateMirror prove this will save time in my coding agent?
No. Its 79-task live intervention is limited to hallucination-affected Python bug-fixing tasks in the evaluated setup. Replay your own tasks and compare task resolution, false blocks, latency, and token use.
Building something that has to hold up?
We do this work for teams — agent reliability hardening, custom MCP servers, and full-stack AI systems built to survive production.