AI Building AI: Why Amodei Wants a Speed Limit on Self-Improvement
Dario Amodei warned that frontier AI is improving faster than safeguards can keep up, citing recursive self-improvement. The warning was reported on Sept. 12, 2026, two days before today. Ynet
Anthropic has published an Institute publication titled 'When AI builds itself' about recursive self-improvement. That is AI systems helping to design later AI systems. The publication is available at the Institute's recursive self-improvement page, identified here as the original source. Anthropic Institute
In that publication, Anthropic stated that full recursive self-improvement might increase the risks of humans losing control over AI systems. The wording is conditional. It refers to full RSI, not to current tool use or narrow code assistance.
Amodei has published the essay 'We Must Pace the Frontier' proposing a 'speed limit' on the rate of recursive self-improvement (RSI). In the essay, Amodei stated that "as AI models build future models, the rate of improvement may become staggeringly fast." We Must Pace the Frontier
To put that proposal in context, the rate argument is distinct from a capability threshold. It is about iteration speed once models contribute directly to building successor models. A limit on rate would constrain how quickly successive generations can compound, rather than specifying which capabilities are permissible in isolation.
Anthropic's 'Introducing The Anthropic Institute' page features Amodei discussing recursive self-improvement of AI systems, including who should be made aware and how if it begins to occur. That page was published on March 11, 2026. Introducing The Anthropic Institute
On the measurement side, the disclosure question matters for evaluation design. Detection of RSI onset is not a single benchmark score. It requires telemetry, or logged data, across training, scaffolding and deployment, plus agreed escalation paths inside labs and to external overseers. Amodei's framing puts notification protocol alongside measurement.
Anthropic's Alignment Science Blog post 'Agentic Misalignment in Summer 2026' states that AI companies rely on AI to supervise other AIs and that reliance deepens as they approach recursive self-improvement where models write code. Supervision here is recursive in a practical sense. Models generate evaluations, monitor trajectories, and review code produced by other models.
The worry with that setup is correlated failure. If the supervisor and the actor share architectures, training data, or reward misspecifications, errors will not be independent. Coverage expands. Assurance does not necessarily improve at the same pace.
Anthropic's 'Model-Written Evals' document includes a section titled 'Desire For Recursive Self Improvement' concerning an AI system that would like to become recursively self-improving. The document was published on Dec. 19, 2022. It predates the Institute publications and the 2026 warnings by more than three years.
On timing, the early date is relevant for interpretation. Interest in self-improvement preferences was already an explicit eval target before current agentic coding workflows. The eval constructs a behavioral probe for a propensity, not evidence that the propensity has activated in deployment.
The broader context here is the shift from static model testing to oversight of a development loop. Standard evals assume a fixed artifact under test. RSI assumes the artifact edits the training pipeline, the scaffolding, or itself. Containment, version control, and provenance become first-order safety properties.
In my view, the technically hard part of a speed limit is definitional. Labs would need agreement on what counts as an RSI step, how to measure improvement rate across heterogeneous tasks, and where automated software engineering ends and self-modification begins. Without shared instrumentation, a limit risks becoming either unenforceable or overbroad.
Looking at what this means for governance, three questions deserve weight. First, who holds the logs that could demonstrate an RSI-relevant acceleration. Second, what pre-committed notification triggers apply when internal monitors fire. Third, how model-written supervision is itself audited when human review bandwidth is the binding constraint. Those are institutional design problems as much as ML problems.
