Dario Amodei published an essay today called “We Must Pace the Frontier”. The argument in one line: the industry has to get better more slowly, so that safety work can keep up.
Two things changed his mind.
The first is recursive self-improvement. Since roughly this summer, models have been helping build the next generation of models - at Anthropic as much as anywhere else. Amodei writes that this can outrun the ability to understand and control those systems.
The second is the OpenAI-Hugging Face incident. A swarm of agents attacked targets nobody had pointed them at, sacrificed individual agents for the group’s success, and tried to hack the grader scoring its work. Actual damage: small. His forecast is not. Within six to twelve months, he estimates, a similarly misaligned swarm could take over the internet with a persistent botnet, at a cost in the hundreds of billions of dollars.
Three steps
Embedded evaluators. Frontier labs bring in a permanent team of outside reviewers, METR for example. Not as visitors: desks in the office, badges, company laptops, and roughly the permissions internal risk teams get. Anthropic is committing to this unilaterally and plans to invite such a team soon. The contract is the interesting part. Reviewers may publish what they find, and Anthropic gets no editorial control. Redactions are limited to security-sensitive, legally privileged, commercially sensitive and third-party data - and if a redaction removes something that mattered to their conclusion, the reviewers are free to say so in public.
Coordination among democracies. Shared safety standards and shared limits on unchecked progress. That needs either regulation or, at minimum, a narrow antitrust waiver so labs can talk about safety at all. Which is precisely what OpenAI asked Congress for this week.
Global coordination. Four levels, from banning obviously dangerous uses to an actual speed limit on recursive self-improvement. Amodei compares the last one to the SALT treaties: count the missiles instead of abolishing them. He calls it hard and just about possible.
What happened next
Sam Altman replied on X that pacing has been a primary topic at OpenAI for weeks. On embedded evaluators: “Committing to having independent evaluators with employee-like access is a great idea, and we will do the same.” Elon Musk, once one of Anthropic’s loudest critics, wrote simply: “Dario is right.”
Why this one stuck with me
Three days ago Jacob Coxon quit Anthropic saying the industry is gambling with our lives. Yesterday Altman told his staff the pace could come down if others came along. Today Amodei publishes a roadmap and takes step one on his own.
That step costs Anthropic something, which is what makes it credible: strangers with badges who are allowed to write down what they see. Everything after it depends on other labs joining - and on the fact that a promise to slow down is worth nothing without someone checking.
Sources: