How to evaluate coding agents on evolving features

Evaluate the whole evolving feature, not just the latest patch. Keep the agent's workspace across requests. After each extension, check both new behavior and earlier promises; then test the combined contract on the final code. Report first-stage, later-stage, and whole-feature outcomes separately, and inspect whether failures were inherited or newly introduced.

Why the last change is an incomplete unit

An extension can satisfy its new request while breaking an older contract. It can also fail because an earlier defect remains in the code. A last-stage-only check cannot distinguish either case from whole-feature success.

FeatureArc built 73 manually reviewed tasks comprising 263 stages from 53 repositories in Python, Go, TypeScript, and Rust. Each task has two to ten stages of one feature. Its construction validates fail-to-pass checks for newly requested behavior and pass-to-pass checks for behavior that should survive, using reference code before and after each stage. It also checks the complete feature on the final workspace.

Across ten configured coding agents in that setup, pooled first-stage success was 62.7% and final whole-feature success was 28.5%. Of 273 runs that passed the last-stage gate, 69 still failed the whole-feature gate. These numbers belong to the selected tasks, agents, harnesses, hidden checks, and execution conditions—not to deployed coding agents generally.

Build a feature-chain gate

Start with real feature sequences, not unrelated tickets. Give each an initial request and at least one extension that reuses earlier code or interfaces. Preserve a short behavioral contract for each stage: what changes now, and what must remain true afterward.

Where a before-and-after reference exists, verify that a new-behavior check fails before the change and passes afterward. Keep acceptance checks independent of the agent's optimistic self-tests where feasible. This is a practical evaluation protocol adapted from FeatureArc, not a controlled finding that these checks automatically improve quality. GitHub's required status checks can enforce a team-owned suite at a branch boundary; the team still must make that suite measure the intended contract.

Record where correctness was lost

Record the first failed stage, whether later work inherited its defect, whether a later edit introduced a regression, and whether the agent noticed and repaired the problem. Compare green self-tests with independent acceptance: a successful test run may test the wrong behavior.

FeatureArc labels inherited defects in 72.9% of failed later stages and reports that agents did not notice an earlier error in 94.9% of stages following a failure. Its annotations were mainly model-judged, with blind human review of a 180-stage sample. Only 52.7% of stages ending with a green self-test passed hidden new-behavior checks. These scoped diagnostic findings are not local release thresholds.

Report first-stage pass, pass after each extension, final whole-feature pass, inherited versus new failure, and elapsed time or tool cost. To compare a human or CI checkpoint with unassisted continuation, run both on matched feature chains. FeatureArc did not test such a checkpoint as a causal remedy.

Scope the conclusion to your workflow

FeatureArc keeps each agent in one session and workspace, with no external failure feedback or rollback and a four-hour budget per stage. Nine evaluated agents used Claude Code while one used Codex CLI, so cross-agent scores couple model and harness effects. Real teams often review, test, correct, and re-plan between requests. Use the study to sharpen measurement design, then set acceptance thresholds from your own repositories and process.

For staged work passed between people or agents, validate the coding specification before handoff. When the changing artifact is the agent's own skills or memory, use a different longitudinal regression test. For promotion, set release acceptance thresholds from local outcome data.