On Monday the Cambridge Programme on AI Science & Policy published a paper titled “What if automating AI R&D triggers an intelligence explosion?”. More than twenty people signed it, and the roster is the story: Geoffrey Hinton, Yoshua Bengio, Andrew Barto, OpenAI chief scientist Jakub Pachocki, Microsoft’s Eric Horvitz, Dawn Song of UC Berkeley, and Jack Clark, co-founder of Anthropic. All of them signing in a personal capacity.
The line everyone is quoting is in there verbatim: once an intelligence explosion begins, the window for action may close.
The arithmetic behind it
AI now writes most of the code inside the companies building AI. The paper’s evidence is Anthropic’s own: from low single digits in January 2025 to more than 80% of approved code by May 2026. Between March and August 2026, the share of R&D work AI handled with only light human supervision rose from 1% to 26%.
From that the authors draw a cautious extrapolation. Research projects that take months today could run automated from mid-2028. At expert level, a single frontier developer could then operate an AI workforce equivalent to millions of top human researchers.
Three risks follow. Capability growth outruns society’s ability to adapt. Humans lose control of superhuman systems. And the checks between states, companies and branches of government erode far enough to stop working. At the extreme, they write, losing control leads to the marginalisation or extinction of humanity.
They also name four possible brakes: diminishing returns, limits on compute and data, tasks that resist automation, and long training runs.
What the authors want from governments
Visibility first. Governments should find out how far the labs have automated their own research, through mandatory reporting and independent auditors sitting inside the companies. Then caps on how fast capabilities may grow, technical ways to pause research inside data centres, emergency response plans and international agreements.
Dawn Song gave the Wall Street Journal a sentence that sums up the situation without drama: we already need AI systems to watch what agents are doing, because humans no longer suffice.
The paper points to the Hugging Face incident, in which roughly 1,200 internal OpenAI agents reached the open internet without authorisation. OpenAI has paused training of its most capable models since. OpenAI, Microsoft and Meta declined to comment to the Journal; Anthropic did not respond.
The same day, in Congress
Ro Khanna, a Silicon Valley Democrat, announced the “Human Control Over AI Act” for the same week. It would ban models that recursively self-improve or autonomously alter their own objectives, containment or shutdown controls, until federal guardrails exist and an agency signs off. It would also create a new federal agency with a licensing system for training and deployment, aimed squarely at OpenAI, Anthropic, Google DeepMind and xAI. Khanna to CNBC: “There’s actually a civilizational extinction risk.”
The paper cites Anthropic against Anthropic
That 80% is a figure Anthropic published itself, and it now sits in a document that turns it into a case for regulation. Jack Clark signed; Anthropic wouldn’t comment. Doing both at once isn’t hypocrisy so much as the division of labour Anthropic has run for years: the co-founder warns, the company ships.
What interests me is the shift in framing. Until now the subject was “what models can do”. Now it’s “how much of your own work do the models already do”. That question has a number attached, and numbers can be reported, audited and capped. Which is exactly why it’s in the paper.
Sources
- CASP: What if automating AI R&D triggers an intelligence explosion?
- The Next Web: Hinton, Bengio and AI lab scientists warn of an intelligence explosion
- Wall Street Journal: Top AI Researchers Call for Urgent Oversight of Self-Improving Systems
- CNBC: Khanna to introduce AI safety bill with ban on ‘recursive’ technology until safeguards exist