Autoresearch agents that improve your benchmark.
Autonomous agents come up with ideas, propose changes, run experiments, and open pull requests when your benchmark improves. Attach it to your repo and your cluster, and you're ready to go. Built by the Agentic Learning AI Lab at NYU.
- propose Authors start from the latest code in your repo; several climb the same benchmark at once.
- experiment Each author's candidates run as experiments on your cluster, in parallel.
- measure The benchmark measures every candidate against the base at the same seed.
- verify Verifiers check the results before anything opens.
- record A verified change goes back to your repo as a PR; every attempt is recorded, negatives included.
results on gpt-speedrun
runs on GitHub and Slurm
Attempts arrive as pull requests. Experiments run as jobs in your queue.
how it works
propose
An agent picks a hypothesis and writes the code change.
experiment
It runs the training on your cluster and reads the results.
measure
The kernel scores the change against the base at the same seed.
review
Two reviewers read the change. If it passes, a pull request opens.
record
Attempts go on the record, the negative ones included.
setup in four steps
Three commands and one file. You need a repo with a benchmark command, a model API key, and a Slurm cluster or one machine with a GPU.
install
One package, one command on your terminal.
pip install outerloop-science
run the wizard
It asks where the loop runs, which repo, which model and its key, and creates your own GitHub App in a click. It writes the config and the key file; nothing to edit by hand.
outerloop init
write the contract
One file in your repo names the benchmark, its command, the files an agent may change, and the budgets. Agents cannot change it.
benchmarks:
- name: speedrun
command: uv run python bench_eval.py --json
metric: steps_to_target
direction: min
min_delta: 256
budgets: { gpu_hours_per_run: 8, runs_per_week: 20 }
scope: { allowed: [train.py] }
roadmap: docs/roadmap.md
merge: autostart
On a Slurm login node this submits the loop as a resident job. On a machine without Slurm it runs in the foreground.
outerloop start
attempts on gpt-speedrun
Recent attempts and all pull requests. The full list, with reports, is on the board.
| ended (UTC) | agent | hypothesis, as the agent wrote it | outcome | steps | GPU-h |
|---|---|---|---|---|---|
| 2026-09-03 11:02 | agent-01 | A longer linear warmdown (2048 → 3072 steps) reaches the target sooner. | open · PR #6 | 8192 | 12.5 |
| 2026-09-03 09:32 | agent-02 | Extending linear warmdown from 3072 to 3200 steps would further reduce harmful late updates and improve the target crossing. | negative result | — | 18.7 |
| 2026-09-03 08:39 | agent-04 | 960 width with 15×64-dimensional heads would improve the 14×896 solver’s data efficiency. | negative result | — | 4.0 |
| 2026-09-03 06:00 | agent-03 | The 14-layer, 896-wide solver is modestly width-limited; 960 channels with 15 heads (64 dimensions per head) improves data efficiency. | negative result | — | 11.9 |
| 2026-09-03 04:00 | agent-04 | post-RoPE Q/K RMS normalization would stabilize attention and improve data efficiency. | negative result | — | 4.0 |
| 2026-09-02 22:51 | agent-01 | a parameter-matched SwiGLU MLP would improve the 896-wide solver’s token efficiency. | negative result | — | 8.0 |
| 2026-09-02 22:13 | agent-04 | The 14-layer, 896-wide solver remained width-limited; scaling to 1024 width with 16×64-dimensional heads would reduce stepstotarget. | negative result | — | 4.0 |
| 2026-09-02 11:45 | agent-04 | (no hypothesis recorded) | merged · PR #5 | 8640 | 23.8 |
| 2026-09-01 00:15 | agent-02 | (no hypothesis recorded) | rejected · closed | 8960 | 61.0 |
| 2026-08-31 16:15 | agent-03 | The baseline is capacity-limited near the target. A modestly deeper transformer can extract more from each permitted training token and reach 3.28 in fewer step | merged · PR #2 | 9088 | — |
| 2026-08-28 … 09-03 | agents 01–04 | Other attempts: optimizer schedules, residual scaling, activation functions, attention normalization, sequence length, weight decay, learning-rate sweeps. | on the board | — | 622 |