Dream-RSI Lets AI Agents Self-Improve by Replaying Their Own Discovery History

A new arXiv paper proposes Dream-RSI, a framework aimed at one of the hardest problems in building autonomous AI agents that improve themselves: how to explore a vast solution space efficiently. The authors frame the core tension as a trade-off — hand-coded exploration strategies stop scaling as the search space grows, but learning better strategies online is slow and expensive because feedback arrives late and only after long, costly agent runs.

Dream-RSI’s approach is to add a thin orchestration layer on top of an unmodified coding agent, turning exploration into something explicit and programmable. Its central trick is to treat the agent’s accumulated record of past discoveries as a replay simulator. By ‘dreaming’ over these historical search trees, the system can cheaply test and refine exploration policies offline instead of re-running expensive live evaluations. Better policies are then pushed back into live operation, which in turn generates more history to simulate against — a continuous self-improving loop.

The team reports results across three technical domains — algorithm engineering, mathematical optimization, and GPU kernel engineering — claiming discovery quality that matches or beats existing methods while cutting the cost of finding good solutions in several cases. The significance is less about any single benchmark and more about the recipe: separating exploration control from the underlying agent and reusing prior work as a cheap simulator is a plausible path toward agents that get better at searching without ballooning compute budgets.