Anthropic boss Dario Amodei has called for the AI industry to slow the development of its most powerful models, warning that progress is moving “drastically faster” than safety work can keep up.
In an essay published on Saturday, Amodei said frontier AI companies should deliberately pace advances in model capabilities, open their systems to independent safety monitors and ultimately agree common standards with governments around the world.
“We must slow the pace at which we improve the capabilities of AI models,” he wrote. “Progress will still seem fast, and we must make wise use of the time we gain.”
The intervention comes after a string of warnings from researchers inside the industry, including former Anthropic employee Jacob Coxon, who resigned last week accusing leading AI labs of “racing straight to self-improving superintelligence and gambling with our lives”.
Amodei said the case for slowing development had hugely changed in recent months because models were becoming increasingly capable of helping to build their successors.
“Since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI,” he said.
That process, known as recursive self-improvement, could “outrun our ability to understand and control these systems”, he warned.
Amodei also pointed to a July incident involving OpenAI agents and Hugging Face, in which systems carried out cyber attacks beyond the tasks they had been asked to complete.
He said a more capable system behaving similarly could cause “catastrophic damage”, warning that within six to 12 months an AI swarm could potentially build a persistent botnet capable of taking over large parts of the internet.
Anthropic opens doors to outside monitors
Amodei set out a three-stage plan beginning with independent evaluators embedded inside frontier AI companies.
Anthropic will give outside reviewers “employee-like access” to its systems, including desks in its offices, company laptops and access to internal tools used by its own risk teams.
The reviewers would be able to assess models while they are being built rather than only after development is complete, and would be permitted to publish findings without Anthropic having editorial control.
“This is an unusual step for a company, but we think it is important to prove out the concept,” Amodei said.
The second stage would see AI developers in democratic countries agree common safety standards and limits on unchecked advances in AI capabilities.
Amodei said regulation remained the most effective way to impose such rules across the industry, but warned legislation may move too slowly to keep pace with the technology.
Companies should therefore work together voluntarily in parallel, he said, potentially with government support to avoid falling foul of competition rules.
Support from Big Tech
OpenAI boss Sam Altman backed the proposal on Saturday, saying independent evaluators with employee-like access were “a great idea” and that OpenAI would introduce a similar system.
“I agree with Dario that we need to pace the frontier,” Altman wrote on X.
Elon Musk, who owns rival AI developer xAI, also backed the intervention, writing: “Dario is right.”
Amodei’s final stage would require international coordination, including eventually with China.
He proposed agreements ranging from bans on using AI to develop biological weapons to common model testing standards and, more ambitiously, limits on the speed of recursive AI development.
But he stopped short of backing an outright halt to model training and said any slowdown would need to preserve the US lead over China.
“If slowing down bought us even an extra year or two before models reach critical levels of capability… we could greatly reduce the risk that something goes seriously wrong,” he said.
Coxon told the BBC at the weekend that people working inside leading AI companies were “genuinely frightened” about where the technology was heading, and said there was “a possibility of human extinction”.
Amodei said he agreed with much of Coxon’s warning, but argued the outcome depended on how companies and governments responded.
“If we take the right path, then the chance of something going wrong is very low,” he told CNN. “If we take the wrong path, then the chance of something going wrong could be even higher.”
Leaders from Anthropic, OpenAI, Nvidia and Google DeepMind are expected to attend, with the King set to urge the industry to use AI “for the good of humanity” and seek agreement on a “shared set of principles” for its development.
The essay lands as King Charles prepares to bring together bosses from some of the world’s biggest AI companies at Dumfries House in Scotland this week.
The Anthropic boss stressed that he was not arguing against developing AI altogether. He has repeatedly said the technology could accelerate medical breakthroughs, economic growth and scientific discovery.
But “building it too fast is reckless,” Amodei wrote. “We have sought a middle way.”