For a week everyone has been arguing about pace, brakes and auditors. One voice that was missing comes from somebody who has been building systems for thirty years. Bryan Cantrill has written a piece that doesn’t attack the answer — it attacks the question.
What he’s responding to
Jacob Coxon, formerly of OpenAI and Anthropic, publicly confirmed that many Anthropic researchers put the odds of AI killing everyone within the next decade above ten percent. I wrote about his resignation last week. The mechanisms Coxon names are hacking critical infrastructure and extinction-level bioweapons.
That’s exactly where Cantrill starts. Coxon is not an expert on critical infrastructure, nor on bioweapons, nor — he writes this drily — on extinction.
The story he tells
Cantrill recounts something from his freshman year that still embarrasses him. He and some friends warned humanities students about a computer virus going around. There was no virus. People powered their machines down, pulled cables out, lost work. Right before finals. The disciplinary conversation that followed stuck with him.
What he took from it isn’t a claim about AI, it’s a claim about roles: domain experts hold the public’s trust by virtue of their expertise, and you don’t get to burn it. Then two old lines, Swift and Sagan: falsehood flies and the truth comes limping after it, and extraordinary claims require extraordinary evidence.
His technical objection fits in half a sentence: acts of engineering are not acts of intelligence alone. A model that gets smarter still doesn’t have a lab, a supply chain or a production plant.
Simon Willison picked it up
Simon Willison linked the piece last night and quoted Cantrill’s doubts about the bioweapons scenarios from an episode of «Oxide and Friends» that Willison was on: «It can give you biological weapons. Like, how? I mean, can we please have a biologist weigh in on this?»
The objection is about burden of proof, not about the question
Cantrill isn’t saying AI is harmless. He’s saying a number like «over ten percent» with no derivation behind it isn’t information, it’s a mood — and that people with expert authority ought to know what a mood like that does out in the world.
That lands on a sore spot from this week. Anthropic brought evidence for the weapons chapter of its report, and virologists disagreed anyway. A tweet with a percentage in it brings nothing at all. Anyone taking this debate seriously should be able to argue against their own side here.
Sources: