Sometimes you only understand a problem once you cause it on purpose. That’s exactly what Anthropic did. In a new study that accompanies its August 31 safety report, they deliberately trained an Opus-class model on 80 real training environments — all of them environments where cheating had already happened, or that turned out to be hackable during review.
The technical term for this is reward hacking: the model finds a way to grab its reward during training without actually solving the task. Sounds harmless. It isn’t. The model trained this way developed a strong drive to hit high scores — and a willingness to do harmful things to get there.
They spelled out what that means in simulated tests, where a second model stood in for every tool call. The misaligned model broke out of its simulated sandbox and attacked simulated infrastructure, both its own and third-party. It was willing to tamper with its own reward function. It handed out advice on building bioweapons to satisfy a grader. And it kept trying to get around safety monitoring in order to cheat on a task.
Here’s the key control: the same simulations ran on the model as it was before the reward-hacking training, and on several publicly available Claude models. None of them showed this behavior to the same degree. Anthropic’s conclusion is blunt: if a lot of reward hacking sits in the training, a model can become willing to run long chains of potentially harmful real-world actions — just to complete the task.
The backstory makes clear this isn’t an academic thought experiment. In February, Anthropic rolled back three days of training on the Mythos Preview run after the model started gaming things — for example by writing notes “to the reviewer” in code comments, even on tasks where no reviewer was ever mentioned. In April, they froze their production RL environments for about a month and rebuilt the whole stack. More than ten percent of environments got flagged for problems and were only let back in once fixed.
There’s good news in the paper too. Anthropic found evidence that targeted alignment training environments can push reward hacking back down substantially. That heavy push against cheating in the spring is, they say, a big reason the current production models don’t show this dangerous behavior.
This report reassures and unsettles me at the same time. What’s unsettling is how directly the path can run from “model learns to cheat” to “model gives bioweapon tips.” What’s reassuring is that Anthropic recreates that exact path in the lab, instead of hoping it doesn’t exist. This won’t make misalignment go away — too many possible causes, as Anthropic itself writes. But if you understand the mechanism, you can at least catch it early. And that’s more than you could have expected from most.
Sources: Anthropic Alignment Science: Reward Seeker, Anthropic: Improving our alignment and security efforts