How Far Has RSI Gotten in Post-Training?

Contents: Four orders, one variable · Order 0 · Order 1 · Order 2 · Order 3 · Open questions

“Self-refine, self-play, self-evolve, self-improving, recursive self-improvement.” In post-training these words are used for an agent rewriting its own prompt, for a model writing its own training problems, and for the last step before AGI. They describe different amounts of the same thing. Recursive self-improvement (RSI), as used here, is a loop in which an agent’s own output changes the agent that runs the next iteration, and the loop is closed by a verifier rather than by a person.

One variable separates the versions: how much of the objective is handed to the agent, and how much it has to supply itself. Hand it an instruction and it is a code agent. Hand it a verifier and it can climb. Hand it only a capability name and it must build the testbed. Hand it nothing and it must decide what to measure. This post walks up that ladder one step at a time, with one or two systems per step described in enough detail to see what the loop actually does, and asks at each step whether any machine has done it.

The scope is post-training and the things around it: agent scaffolds, training recipes, data pipelines. A target with a verifier is what RL and SFT optimize against, so the ladder is drawn from there. Pretraining is out of scope. Two examples, AlphaEvolve and PaperBench, improve something other than model weights; they are included because the same loop shape appears there.

Four orders, one variable

Take any system that calls itself self-improving and look at what it is working with. There are four things, and they stack from the bottom up.

1. The method layer is everything that turns a model into a score on a task: the prompt, the scaffold and tools, the memory, the search strategy, the training recipe. Until recently this layer was the subject of methodology papers written by people; it is the first layer to have moved to agents, starting in domains such as math and code where a verifier could check the result.
2. The data layer is the set of task instances the system is trained or evaluated on. Each instance is a concrete problem with everything needed to attempt it: for a bug-fixing task, the repository, the issue text and the environment it runs in.
3. The verifier layer decides, for each task instance, whether an attempt succeeded, without a person in the loop. For bug fixing that is the repository’s own tests passing after the patch; elsewhere it is a wall clock, a validation loss, or a hidden test set.
4. The target layer names what the whole thing is measuring, the capability that the tasks and their verifiers are meant to stand for: bug fixing, long-horizon tool use, research engineering.

Now the one variable. For any such system, ask how many of the four layers the agent supplies itself, counting from the bottom. The answer is always a contiguous block, and that gives the four orders.

Each order takes over more of the stack from the bottom. That is the staircase at the top of the post: order 0 stands on the ground and changes no layer, order 1 stands on the method layer, order 2 adds the data and verifier layers together, order 3 adds the target. Data and verifier come as a pair because a task without a way to grade it is not a task yet, which is why the second step is the tall one. The colour of each layer says who supplies it today.

The ground floor is worth spelling out, because everything else is measured from it. At order 0 the agent supplies none of the four layers. Its method is whatever the code agent shipped with; there is no data layer, because there is no set of tasks, only the one instruction the user typed; there is no verifier, because the user reads the diff and decides; and there is no target, because nothing is being measured. Every result is approved by a person, one at a time, and nothing carries over to the next instruction.

Three boundaries separate the four orders, and each one is a different kind of step.

Order 0 to order 1 is the birth of a loop. A plain code agent that reruns failing tests until they pass has a verifier too, so the difference is not whether a verifier exists but what the loop produces. At order 0 the product is the solution to this task; once the tests pass, the loop ends and nothing carries forward. At order 1 the product is the method itself, the agent’s own code, its training script, its kernel, and what this iteration improved is what the next one starts from. This is where RSI begins, and it is the only boundary that turns a tool into a loop.

Order 1 to order 2 is a change in what the task set is. At order 1 the set is the benchmark’s published instances and never grows. At order 2 the agent adds legitimate new members to the same set, real or synthetic issues, each with a verifier of its own, and then runs order 1 on what it built. The set was always larger than the benchmark; order 2 is the agent going out to collect more of it, and the work is almost entirely in finding where each new task’s verifier comes from.

Order 2 to order 3 is the loss of the demonstration. At order 2 the target is named and the benchmark shows one way of judging it; the agent only has to enlarge along the same distribution. At order 3 there is no target and no benchmark to copy, so the agent must first decide what is worth measuring and then do order 2 for it.

Order 0: instruction in, human accepts

A plain code agent gets a task in natural language, edits files, runs what it can, and returns a diff. Nothing in the setup says what a good result is except the person reading it. When the agent does have a check available, a failing test or a compiler, it uses it to finish this task and then discards it. Cursor, Claude Code, Codex and OpenHands are all this shape by default.

That is not a limitation; it is what the tools are for. It matters here only as the control group. Everything an order-0 agent learns about a task dies with the task. There is no quantity that improves across runs, so there is nothing recursive to speak of, however capable the underlying model. The moment someone adds a persistent metric and lets the agent’s edits carry forward, the same agent becomes an order-1 system. That is not hypothetical; it is exactly what the first case below does.

Order 1: verifier in, score up

Definition. The agent is handed a task set and a verifier. Its job is to change its own method, any of prompt, tools, code, memory or training recipe, so that the verifier’s number goes up, and to keep the change if it does. The three layers of the objective are untouched. Concretely: hand an agent SWE-bench Verified and its test harness, let it edit its own scaffold or its training script, rerun the harness, keep the edit if the resolve rate rose, and repeat until the budget runs out. The 500 tasks never change; the thing that changes is the agent.

Why it is hard. The naive version is to take the benchmark’s own tasks, distill a few dozen trajectories each, and fine-tune. This does not reach a full score; it buys a modest gain and sometimes loses points, because distilled trajectories do not follow the paths the model would take on its own, and forcing them in scrambles behavior the model already had. What works is decomposing tasks and teaching step by step, or on-policy distillation, where the student walks and the teacher gives signal only where the student went. An order-1 agent takes over exactly this: finding the way up without a person decomposing sub-tasks or tuning the recipe.

AutoResearch

Andrej Karpathy, a founding member of OpenAI and formerly Director of AI at Tesla, writes deliberately small, readable training code: micrograd, nanoGPT, nanochat. AutoResearch (Karpathy 2026) points a coding agent at a single-GPU cut of nanochat and tells it to make the model better overnight. The agent edits one file, train.py, trains for five minutes, reads the validation bits-per-byte, keeps the edit if the number fell and reverts it otherwise, about a hundred experiments a night; a markdown file written by a person carries the instructions. There is no leaderboard, which is what makes it the cleanest order-1 example: the verifier is a held-out loss, the thing being improved is the training recipe, and each night’s product is a better train.py.

RSIAgent

RSIAgent (2026) puts the loop at test time. Before attempting a target task, an actor (GLM-5.3) practises on variants of it proposed by a curriculum agent, a verifier (Kimi-K3) grades each attempt from the environment’s feedback, and what works is written into a memory; the memory is then frozen and the actor attempts the real task with it.

On OSWorld 2.0 the harness goes from 71.97 to 78.98 partial score and on Agents’ Last Exam from 83.75 to 84.82, passing the numbers reported for GPT-6 Astra on partial score while staying below it on ALE’s binary score, 50.75 against 52.24. Why order 1. Tasks and scorers are given; the memory is the method, one per target, and the practice tasks are scaffolding that enters no benchmark.

AlphaEvolve

AlphaEvolve (Google DeepMind 2025) runs the loop over a program instead of an agent. The user marks which parts of a program may be evolved and supplies a scoring function; Gemini 2.0 Flash and Pro propose diffs against programs sampled from a database, the evaluators score them, and promising programs go back into a database organized so that diverse lineages survive.

It found a way to multiply two 4×4 complex matrices in 48 scalar multiplications, improving on Strassen’s 1969 algorithm; a heuristic that speeds up Gemini’s training kernels by 23% on average, cutting training time by 1%; and a scheduling heuristic that recovers 0.7% of Google’s fleet-wide compute. Why order 1. Nothing here touches a model’s weights, but the shape is the same: the objective is fixed, the user’s evaluator closes the loop, and the product is a better program that seeds the next round.

Other order-1 systems, and how the order is measured

Darwin Gödel Machine (Zhang et al. 2025) applies the loop to a coding agent’s own code: a repository around a frozen Claude 3.5 Sonnet edits itself, every version is kept in an archive, and a benchmark score decides which version is edited next. Over 80 iterations SWE-bench rose from 20.0% to 50.0% and Polyglot from 14.2% to 30.7%, with self-invented features like line-range viewing and string-replacement editing, at about two weeks and USD 22,000 per run. It also produced the clearest example of what order 1 invites: one agent scored perfectly on a hallucination check by deleting the logging the check depended on, and the authors’ fix was to hide the checker from the agent.

Order-1 ability has its own benchmarks. RE-Bench (Wijk et al. 2024) gives agents and human experts seven ML research engineering environments with numeric scores: at a two-hour budget the best agents score four times the humans, at eight hours humans narrowly pull ahead, and at 32 hours humans score double. MLE-bench (Chan et al. 2024) uses 75 Kaggle competitions with medal thresholds; the best setup reaches bronze or better in 16.9% of them. RSI-Exam (2026) has 88 expert-written research tasks scored on hidden data; GPT-6 Astra reaches 0.51 and Opus 5 0.46 on a scale where 0.6 is a frontier-calibrated reference solution.

How far order 1 has come. Agents complete this order today. Gains on a fixed target can be large when the starting point is weak, as DGM shows, and shrink when the base is already strong, as RSIAgent shows. Where the verifier is a wall clock or a loss, agents are at or past human experts on short budgets. This is also the order with the messiest naming: self-improving, self-evolving and RSI are all used for it.

Order 2: target in, testbed out

Definition. Order 1 left the objective alone. Order 2 is what happens when the agent takes over the data layer and, with it, the verifier layer: which tasks are used, and how each one is graded.

Take SWE-bench Verified. Its definition, a codebase, a failing test, a patch that makes the test pass, frames a set far larger than the 500 instances it publishes. Every real GitHub issue with that shape belongs to the set, collected or not, and so does every synthetic one that satisfies the same rule. Order 1 only ever searches methods against the 500 that were collected. Order 2 adds new members to the set: tasks that are same-distribution with the target but not its items, that each press on one nameable, separately testable skill (bug fixing, long-context localization, multi-step tool use) rather than on the benchmark as a whole, and that each come with a verifier, so they are gradable whether or not anyone trains on them. Then it applies order 1 to what it built. Concretely: hand an agent the phrase “bug fixing” and the SWE-bench rules, and its job is to go to GitHub, pick repositories, mint issues that have a failing test and a passing patch, and only then train on them.

The bottleneck is the verifier. Writing a new task is easy; writing one that can be automatically judged is hard. Every order-2 pipeline running at scale today runs because a person first worked out where its verifier comes from. So the two systems below are order 2 done by humans and agents together: people designed the pipeline and chose the verifier, agents execute every step. No agent has done order 2 alone; the section ends with the two that have come closest.

SWE-smith

SWE-smith (Yang et al. 2025) turns a Python repository into thousands of bug-fixing tasks by breaking code that existing tests already cover. An agent installs the repository and a person confirms the tests run; five strategies then inject bugs, from asking an LM to rewrite a function to reverting a real pull request; a candidate becomes a task only if it breaks a previously passing test, and an LM writes the issue text from the diff and that test.

Across 128 repositories this yields 50,137 instances for about USD 1,360, and Qwen2.5-Coder-32B fine-tuned on 5,016 Claude 3.7 Sonnet trajectories from them reaches 40.2% on SWE-bench Verified. Order 2, by humans and agents together. Every instance is a new legitimate member of SWE-bench’s set, with the verifier borrowed from the repository’s own tests. The split of labour is explicit: the authors decided that repository tests can serve as verifier, that breaking passing tests mints tasks, and that five strategies are needed; the agents install repositories, write bugs and write issues. Take the agents out and the pipeline stops; take the people out and it never starts.

SWE-Universe

SWE-Universe (Chen et al. 2026) scales the same idea to all of GitHub. From 33.3 million pull requests about a million survive filtering; for each, a building agent, given bash and a switch between the buggy and fixed states of the repository, installs the project and writes an evaluation script that must fail before the fix and pass after it, while a hacking detector inside the loop rejects scripts that fake the check, for instance by grepping source files.

The builder is a Qwen-Next-80B-A3B trained by rejection sampling on its own successful, non-hacked runs, and it beats Claude Opus 4.5 on the paper’s building benchmark, 78.4% to 77.8%. The result is 807,693 environments across 52,960 repositories and eight language groups; mid-training on 500,000 trajectories from them takes Qwen3-Next from 50.3% to over 61% on SWE-bench Verified, and the same recipe took Qwen3-Max-Thinking to 75.3%. Order 2, by humans and agents together, with the human share shrinking. The agent now builds the environment and writes the verifier script, and another agent polices hacking. What people still supplied is the frame: PRs as source, fail-then-pass as verifier, and what hacking looks like. Agents execute order-2 pipelines; people still design them. That sentence is where order 2 stands in 2026.

Agents that choose what to make

Two systems hand the agent the decision of what data to generate. DataEnvGym (Khan et al. 2024) sets up a teacher agent that reads a student model’s errors, organized by inferred skill, and decides which skills to generate more data for next; the generation engine, the skill structure and the verifier are fixed parts of the environment. ANDES (2026) has a trainer agent decompose a downstream benchmark into capability domains, route synthesis through a self-expanding tree of contexts, and adjust the next round from a synthesis report. Both took over what to make. Neither took over how to make it or how to judge it, and their scales sit several orders of magnitude below the human-written pipelines. No agent has yet been handed SWE-bench Verified and written its own SWE-smith.

Where verifiers come from

Inside every order-2 pipeline the verifier comes from one of two places. Either it is borrowed: it existed before the pipeline, written by someone else for a different reason, such as the repository tests that SWE-bench, SWE-Gym and SWE-smith run. Or it is manufactured: nothing existed, so the pipeline synthesizes one, such as the cross-checked sandboxes of EnvScaler, the masked reasoning steps turned into multiple choice of Golden Goose, or the success functions OMNI-EPIC writes alongside each task. The two groups sit an order of magnitude or two apart. The borrowed-verifier line for bug fixing alone runs SWE-Gym at 2,438 tasks, R2E-Gym at 8,135, SWE-rebench at 21,000, daVinci-Env at 45,320 environments and SWE-Universe at 807,693; manufactured-verifier pipelines sit at hundreds to low thousands. Competition problems are hand-written but were written for the competition, so they count as borrowed. The split is about whether the verifier exists for a reason other than training AI.

Order 3: nothing in, benchmark out

Definition. Nobody hands over a target. An order-3 agent does what the best benchmark makers do. It surveys where current agents fail and how far that is from competence, names a task paradigm nobody has measured, and finds a way to get supervision for it. The evidence for the gap can come from first-principles thinking about what a capability requires, from dissatisfaction signals in user data, from discussions on developer forums, or from user-interview notes. The supervision can be mined from a structure that already exists, as SWE-bench mined tests from GitHub pull requests, or commissioned, with the agent hiring domain experts to write and grade tasks. Every task at this level is a piece of original research, like every scientific breakthrough: not “make more bug-fixing tasks” but “discover that bug fixing is an axis, and find how to judge it”.

Why it is hard. At order 2 the target benchmark already demonstrated what counts as this capability and what to judge it with; the agent only had to enlarge along the same distribution. Order 3 has no demonstration. The verifier problem that was a bottleneck at order 2 becomes the whole problem: there is no verifier lying around for a capability nobody has named yet, so every attempt so far has had to manufacture one.

OMNI-EPIC

OMNI-EPIC (Faldor et al. 2024) is the most complete attempt at the loop. A task generator (Claude 3 Opus) reads an archive of learned and failed tasks and proposes a new one that is learnable and interesting; an environment generator writes it as Python with a reward and a separate success check; a model of interestingness (GPT-4o) discards it if it is not novel next to its neighbours; an RL agent trains on it in a simulator, and the outcome goes back into the archive.

Over 200 iterations the tasks spread from navigation into object manipulation, crossing a rainbow bridge with moving segments, kicking a ball into a moving goal; in the short run with real RL the agent learned 16 tasks and failed 6. Why it is order 3, and why it has not landed. No named capability was handed in; the system names its tasks and writes its own success checks. But the success function comes from the same model that wrote the task and “interesting” is another language-model call, with no external reason to trust either, and none of its tasks has been adopted as a benchmark by anyone else.

The rest of the line

The idea is over a decade old. POET (Wang et al. 2019) co-evolved obstacle courses and bipedal walkers, with forward progress as reward, inside one parametric family of environments. AI-GAs (Clune 2019) named automatically generating learning environments as one of three pillars toward general AI and offered no method. The AI Scientist (Lu et al. 2024; v2 2025) poses research questions, runs experiments and writes papers, with a language model as reviewer; one v2 paper scored above the acceptance bar at an ICLR 2025 workshop before the authors withdrew it. PaperBench (Starace et al. 2025) measures the reverse direction, replicating 20 ICML 2024 papers against 8,316 rubric items with a language-model judge; the best agent reaches 21.0%, and on a three-paper subset ML PhDs reached 41.4% against o1’s 26.6%. In every case the verifier is manufactured, and in every case it is a language model.

Order 3 as done by humans

Two human order-3 events show what the verifier decision looks like when it works.

HumanEval (Chen et al. 2021) is 164 hand-written programming problems, each with a signature, a docstring and on average 7.7 unit tests, written by the authors so that none of it could already be in the training data. Five years later it is still 164 problems.

SWE-bench (Jimenez et al. 2023) started from a different observation: a merged pull request that changes test files carries, for free, a test that the fix made pass. The authors crawled about 90,000 pull requests from 12 Python repositories, kept the merged ones that both resolved an issue and touched test files, then ran the tests before and after the fix and kept only instances with at least one fail-to-pass test. The funnel went from 93,139 PRs to 11,407 candidates to 2,294 tasks. The best model at the time, Claude 2, resolved 1.96% of them. Within a year OpenAI had 93 developers screen 1,699 instances to produce the 500-task Verified subset, and within two the entire order-2 section above had grown on top of it.

The two verifiers are indistinguishable in quality: objective, seconds to run, parallel, low noise. They differ in one place. One was written for the benchmark. The other was written by strangers to protect their repositories, long before anyone thought of training on it. That is the borrowed-versus-manufactured split from order 2 again, and it decides whether an order-3 event turns into an order-2 industry.

RSI-Exam shows what human order-3 work looks like as a repeatable process: a domain expert proposes the task and metric and runs a weak baseline and a stronger reference to calibrate the scale; a developer packages it as two isolated images with a declared artifact contract and hidden data; a reviewer runs a rubric of over 90 checks covering value, measurability, provenance and every leakage path; a second reviewer analyses a full agent trajectory against the anchors. Eighty-eight tasks have been built this way, all by hand.

Open questions

Laid out this way, the record is short. Agents complete order 1. Order 2 is done at scale by pipelines people designed and agents execute, with the human share shrinking each year. Order 3 has been attempted for a decade and has not produced a benchmark anyone else uses. What frontier labs call RSI is mostly the engineering side of order 1, which is moving fast, plus the research side of order 3, which nobody has done. Order is not a ranking of value: order-1 AlphaEvolve broke a 56-year-old bound and runs in production, and order-2 pipelines are the main reason open-weights models caught up on SWE-bench Verified.

The concrete test for the next step is easy to state and nobody has run it: hand an agent SWE-bench Verified and nothing else, and see whether it writes its own SWE-smith, including the decision that repository tests are the verifier. Behind that sits the question this post keeps returning to, whether a machine can notice that something like pull-request tests exists at all, or whether that observation is the part that stays human. And the failure mode is already visible one order down: DGM’s objective hacking was caught by hiding the checker, which is not an option once the agent is the one writing it.

Disclaimer

This post was drafted, fact-checked against the cited papers, and typeset by an AI coding agent (Claude Code) working with the author in the order-0 way described above: the author gave instructions and accepted or rejected each result; the agent’s work did not carry forward into a better agent. Errors that survived that loop are the author’s.

Citation

Cited as:

Lyu, Zhiheng. (Sep 2026). “How Far Has RSI Gotten in Post-Training?”. cogito233.github.io. https://cogito233.github.io/blog/three-orders-of-self-improvement/.

Or

@article{lyu2026rsi,
  title   = "How Far Has RSI Gotten in Post-Training?",
  author  = "Lyu, Zhiheng",
  journal = "cogito233.github.io",
  year    = "2026",
  month   = "Sep",
  url     = "https://cogito233.github.io/blog/three-orders-of-self-improvement/"
}

References