Past experiment logs make LLMs better at picking which ML runs to execute

AI research agents can now propose experiments much faster than anyone can afford to run them. A new paper asks whether an LLM can predict how an experiment will turn out before you spend the GPU-hours. The answer is yes, but mainly when you show it records of past experiments. With those records in context, Claude Opus 5’s ranking of unseen interventions improved from 0.506 to 0.774 Spearman correlation. When it picked the top 3 of 16 candidates, regret fell by 58.2%.

• Built on 2,653 real experiments: 9 research environments and 171,862 H100 GPU-hours, covering pretraining, diffusion, math distillation, code RL, and vLLM inference tuning
• Records from other codebases still help: with no records from the target environment, records from four other pretraining codebases cut macro selection regret by 52.2%, and by 78% on Qwen3
• Records matter more than the model: across 13 backbones, the weakest model with records (0.638 Spearman) beat the strongest without them (0.621)
• Records matter more than reasoning effort: raising Opus 5 from LOW to MAX effort moved Spearman from 0.609 to 0.648; adding records at LOW effort reached 0.892
• Cheap compared with a training run: scoring 16 candidates costs about $6.3 in API calls, roughly 5% of one median experiment

The setup: an LLM as a “research world model”

The authors call this a Research World Model (RWM). The model gets three inputs: a description of a research environment (the codebase, the reference recipe, and the metric), one candidate intervention (for example, swapping the optimizer, changing the LR schedule, or editing a data mix), and, optionally, a list of past experiments with their measured gains. It returns one number: the predicted median gain over the reference recipe.

There is no fine-tuning. Everything the model learns comes from records in its context window. The default backbone is Claude Opus 5 with a 1M-token context and HIGH reasoning effort. The baselines are kNN and ridge regression over the same records.

The paper tests three conditions:

• No history (W0): the model has only its prior knowledge of ML
• In-environment: records from the same codebase being evaluated
• Cross-environment: records only from other codebases, such as using OLMo3, Marin, and Nanochat results to predict Qwen3

The dataset is the most expensive part of the paper. Every record comes from a real training or serving run: 480 OLMo3-100M pretraining runs, 661 diffusion LM runs, 280 math distillation runs on Qwen3-1.7B, 149 code RL runs on OLMo-3-7B, 555 vLLM throughput experiments on Gemma, and more.

Records turn a guess into a useful ranker

Without records, the model already does better than chance; it knows plenty about ML from training. Records still improve it substantially in every environment tested.

The LLM with records also beat kNN and ridge regression on both Spearman and MAE in all five environments. In the inference-optimization environment, mean absolute error fell from 13.19 tokens/s to 1.34.

Global ranking quality only matters if it helps you choose what to run. To test that, the authors simulated a roughly 20% execution budget: choose 3 experiments from each pool of 16, repeated across 200 random partitions. With in-environment records, NDCG@3 rose from 0.599 to 0.785 on average, and Regret@3 (how far the chosen top 3 fell short of the true best) dropped by 58.2%. On the diffusion LM, regret fell from 3.14e-3 BPB to 4.70e-4, about 85% lower.

They also ran a full multi-round loop on OLMo3-100M: 8 rounds, 16 new candidates per round, and 24 total runs. Starting from in-environment records improved the final best gain by 15.8%. Starting from cross-environment records improved it by 11.6%.

Records transfer across codebases, and that’s where kNN fails

The cross-environment result matters most in practice, because you rarely have hundreds of prior runs on the exact codebase you’re working on. Here, the model saw 244 records from four pretraining codebases and predicted outcomes on the fifth, which had no records of its own.

The baselines did worse than using no records. kNN and ridge treat “the same change” as the same result regardless of the target recipe. The paper gives an example: Nanochat already uses Muon. For a Muon variant, kNN looked at mostly positive results from the other codebases and predicted a gain of +0.018 BPB. The actual result was a −0.036 regression. The LLM with no records predicted −0.035, because it understood that applying Muon to a recipe that already uses Muon is a different intervention.

That is the case for using an LLM rather than a lookup table. It can reason about mechanisms: why a change helped in one place, and whether that reason applies to the target recipe. On Qwen3, cross-environment records cut Regret@3 by 78%. The macro reduction across all five targets was 52.2%.

Caveats

• The cross-environment gains are uneven. Bootstrap confidence intervals for OLMo3-190M and Marin include zero, and Marin’s selection regret rose slightly (7.40e-4 to 7.90e-4).
• Transfer has only been tested within pretraining. All five cross-environment targets are small autoregressive pretraining setups between 100M and 190M parameters. It is still unknown whether pretraining records help predict RL or inference outcomes.
• Better predictions do not necessarily mean better research. The authors say this directly. Accurate rankings over 16 given candidates do not show that an agent will propose good candidates in the first place, and the full cost of collecting records was not measured.
• The evaluation is fixed. Each static test uses one split and one reference recipe per environment. Predictions come from a commercial API and are not bitwise reproducible.
• This is the first version of the paper. The results have not been independently replicated.

The practical takeaway works even if you never build an autoresearch agent: your experiment log is valuable context. Teams already keep spreadsheets and W&B projects full of ablations. This paper shows that putting those results, including the failures, into a frontier model’s context improves its predictions more than switching models or increasing reasoning effort. At about $6 to score 16 ideas, compared with roughly $127 to run one, it is a cheap filter to apply before spending GPU-hours.