Why One AI Leader Says Smart Computers Need to Slow Down
Dario Amodei warned that top-end AI is getting better faster than safety checks can keep up, because of recursive self-improvement. That warning was reported on Sept. 12, 2026, two days before today. Ynet
Anthropic has published an Institute piece called 'When AI builds itself' about recursive self-improvement. That is AI helping to build the next version of AI, like a printer that builds a faster printer. The piece is on the Institute's recursive self-improvement page, used here as the original source. Anthropic Institute
In that piece, Anthropic said full recursive self-improvement might raise the risk that people lose control over AI systems. The word might matters. It is about full RSI, not normal tool use or narrow help with code today.
Amodei also wrote an essay, 'We Must Pace the Frontier,' calling for a 'speed limit' on how fast RSI happens. In it he said, "as AI models build future models, the rate of improvement may become staggeringly fast." We Must Pace the Frontier
To put that call in context, it is about speed, not a single cutoff. It asks how fast new versions can build on each other once models help make the next models. A speed limit would slow how fast those steps pile up, not list which features are allowed on their own.
Anthropic's 'Introducing The Anthropic Institute' page has Amodei talking about recursive self-improvement, including who should be told and how if it starts to happen. That page came out March 11, 2026. Introducing The Anthropic Institute
On tracking, telling others matters as much as measuring. The start of RSI would not show up as one test score. It would need logged records from training, support software, and live use, plus clear steps for alerting people inside labs and outside overseers. Amodei's framing puts that alert plan next to measurement.
Anthropic's Alignment Science Blog post 'Agentic Misalignment in Summer 2026' says AI companies use AI to watch other AIs, and that use grows as they get closer to RSI where models write code. In practice, models make tests, watch work, and check code written by other models.
The worry with that setup is shared blind spots. If watcher and worker share the same design, training data, or flawed goals, their mistakes will line up. More checking happens. Safety does not get better at the same speed.
Anthropic's 'Model-Written Evals' paper has a part called 'Desire For Recursive Self Improvement' about an AI system that would want to become recursively self-improving. It came out Dec. 19, 2022. That is more than three years before the Institute pieces and the 2026 warnings.
On timing, that early date helps with reading. Researchers were already testing for interest in self-improvement before today's coding helpers. The test looks for a tendency. It is not proof the tendency is active in real use.
The broader context here is a move from testing a fixed product to watching a loop that changes itself. Normal tests assume the thing being tested stays still. RSI assumes it edits its training process, its support tools, or itself. Tracking versions, containing changes, and knowing origins become core safety jobs.
In my view, the hard part of a speed limit is definition. Labs would need to agree on what counts as one RSI step, how to measure speed across different jobs, and where normal coding help ends and self-change begins. Without shared measuring tools, a limit could be impossible to enforce or far too broad.
Looking at what this means for governance, three questions deserve weight. First, who keeps the logs that could show an RSI speedup. Second, what agreed alarms require a notice when internal monitors go off. Third, who checks the AI checkers when people lack time to review everything. Those are organization problems as much as technical problems.
