Self-improving agent harnesses overfit. How Google fixes it

Letting an agent rewrite its own harness works, until you test it on a benchmark it didn’t train on. Google Cloud AI Research ran four recent harness-evolution methods on the same setup: the method that scored best on the tasks it evolved against gained less than a point anywhere else, and two of the four ended up worse than the harness they started from. Their fix, RRSI, treats harness evolution like model training and regularizes it. It posts the smallest evolve-set gain of any method and the biggest held-out gain.

• Unregularized evolution: 92.8 on the evolve set, 40.3 on held-out benchmarks. The unevolved harness scores 39.7 held-out, so nearly all of the gain was memorization.
• RRSI: 90.5 evolve, 43.6 held-out, the only method to beat the starting harness by more than a point out of distribution
• 2.42M policy tokens per trial vs. 3.80M for unregularized evolution (36% fewer)
• Terminal-Bench 2.1 goes from 74.2 to 80.2 with Claude Opus 4.8, and the harness carries +1.8 to SWE-bench Verified, which it was never scored on
• No held-out split regresses in any of the three domains

The overfitting problem

A harness is everything around the frozen model: prompts, control flow, tools, memory, context management. Recent methods like Meta-Harness and AHE (and Self-Harness, which we covered in June) automate harness tuning. An LLM proposer reads failed trajectories, edits the harness, and keeps the edit if the score goes up.

The catch is that the same finite eval set gets reused every round, and each round’s proposals depend on the last round’s scores. That’s adaptive overfitting, the same problem as tuning hyperparameters on your test set. The paper names three ways it shows up: benchmark-specific fitting (hard-coding task details), noise chasing (keeping edits that won by luck on a stochastic eval), and complexity accumulation (bolting on steps that cost tokens without adding a real capability).

Table 1 of the paper makes it concrete. Every method starts from the same harness, policy, evolve set (120 Harvey LAB legal tasks), and candidate budget, then gets tested on three benchmarks it never saw: JobBench, GDPval, and APEX-Agents.

The ranking flips. The more a method gained on the evolve set, the less of it survived outside it.

How RRSI regularizes the search

RRSI leaves the edit space fully open: the proposer can still touch prompts, control flow, tools, skills, memory, and subagents. Instead of limiting what can change, it limits how feedback turns into permanent changes, on both sides of the loop.

On the proposal side:

• Edit budget (L0-style). Each candidate may bundle only a limited number of independent edits. The budget anneals on a cosine schedule: early rounds can make several coordinated changes, late rounds make one or two, so every gain can be traced to a specific edit.
• Edit history. Every candidate’s component, hypothesis, diff, score change, cost change, and accept/reject decision gets logged. The proposer sees this history, so it stops retesting ideas that already failed.
• Forced exploration. If progress stalls inside the noise band for a few rounds, part of the budget goes to component types the search hasn’t touched yet (for example, when it has only been rewriting prompts).

On the selection side:

• Leakage critic. Before any evaluation, an LLM critic reads the diff and rejects anything encoding task names, entity names, answers, or other benchmark-specific logic. A leaky edit never gets scored, so it never gets a chance to look good.
• Noise floor. Before evolving, RRSI runs the unchanged base harness repeatedly to measure eval noise (δ). A candidate must score at least best-so-far minus δ. This stops the search from sliding downhill through a chain of small regressions that each look like noise.
• Cost-aware acceptance (L2-style). A gain bigger than the noise band is only accepted if the added token cost fits a budget that grows with the gain (ΔC ≤ β₀ + β₁·ΔS). More compute has to buy more score.
• Pruning (L1-style). Components that produce no positive gain over a pruning window get flagged for deletion. A mechanism has to keep earning its place.

The paper’s case studies show the rules working. In the first coding round, two nearly identical candidates both added a pre-submission verification step. The one with a clear gain was accepted. The one with a smaller gain and higher token cost was rejected by the cost rule. In round 8, a candidate that re-checked the original task spec before submitting was cheaper, but still got rejected because its score fell below the noise floor. A small, generic fix for a recurring “workdir must be an existing directory” tool error was accepted.

What each piece buys

The ablation isolates both halves. Removing either group of regularizers raises the evolve score and lowers transfer:

The acceptance rules do the most for cost: without them, tokens per trial rise by half and most accepted edits go to noise and extra context. The proposal rules matter even though they reject nothing: dropping them costs just 0.2 points on the evolve set but 1.7 out of distribution.

The results also hold outside judge-graded tasks. EngDesign and Frontier-Eng are graded by deterministic simulators, not an LLM judge, so the harness can’t win by writing what a judge likes. RRSI still gains +4.9 on EngDesign and +4.3 medal points (24.3% relative) on Frontier-Eng. The harness also transfers across models. Evolved with Gemini 3.5 Flash, it lifts Terminal-Bench 2.1 from 64.6 to 78.7, and the same harness run unchanged on the much smaller Gemini 3.1 Flash Lite goes from 11.2 to 14.6.

Caveats

The absolute out-of-distribution gains are modest: 1.8 to 4.7 points. RRSI still costs more than no evolution at all (2.42M vs. 1.56M tokens per trial, 26.3 vs. 21.2 steps), so part of the gain is bought with test-time compute. “Out of distribution” here means a different benchmark in the same domain (legal evolve set, office-work test sets), not a different kind of work. On Frontier-Eng, only 38 of 47 tasks could be built in their sandbox. The method also adds hyperparameters (edit budgets, noise band, β₀, β₁, pruning window) that are tuned on the evolve set itself.

The lesson carries over to any self-improving loop: if you score candidates against the same finite set every round, you are training on your test set. Measure your eval noise before you start, make every edit pay for its cost, and delete what stops earning its place.