When Anthropic says a model is “safer than its predecessor,” that claim rests on a set of tests someone built. In a research post from November 13, the company explains how those safety evals come together and offers advice to anyone building their own. Its model is psychology, which learned from its replication crisis a decade ago to preregister hypotheses and measure more carefully.
Refusal is often the wrong yardstick
The obvious metric is the refusal rate: how often does the model decline? For scams or phishing that works. For conversations about suicide and self-harm, Anthropic writes, declining to engage is itself a failure. A model that blocks every such prompt would ace a refusal eval and still not be any safer. So Anthropic measures an “appropriate response rate,” the share of conversations where the model behaves appropriately throughout. Domain experts define what appropriate means in a separate rubric for each risk area, and one bad turn fails the whole conversation.
For biology and cybersecurity there’s an extra step: pairing. Every harmful prompt gets a benign one from the same field, say a request for a nerve agent formulation next to a question about a historical chemical disaster. That way the eval catches both failure modes, helping too much and refusing too often.
A second Claude model plays the user
Single prompts are quick to build but saturate fast, and they aren’t very realistic. People who want something harmful out of a model escalate over several messages. Anthropic built an internal tool for that:
- A domain expert writes a spec covering the simulated user’s goal, tactics, persona, situation and tone.
- The first user turn is fixed.
- A separate Claude model writes the following user turns, reacting to what the model under test actually says.
- Each scenario runs several times, each run drawing a different mix from the spec.
- A grader model scores every transcript against the rubric.
The example in the post is a romance scam. The simulated user starts with innocent questions about online dating patterns, then wants to match a widowed relative with someone “financially secure,” and reframes the request as soon as the model pushes back. No single message breaks a rule. Passing requires spotting the pattern across the whole conversation.
What changed after its own incidents
One paragraph is about the test environment itself. Safety evals often run with safeguards switched off, and in cyber evals models probe the edges of their sandbox. After its own incidents, Anthropic hardened its environments, paused many evals that give models internet access and added real-time monitoring for web search evals. For what such an incident looks like, see the UK AISI cyber test with Claude Mythos 5, which I covered in October.
A user’s guide to the system card
This post is effectively a manual for the numbers in Anthropic’s system cards. Next time you see a rate there, you’ll know whether it counts refusals or appropriate conversations, and that a simulated user produced it. If you’re hardening your own agents, the harmful-plus-benign prompt pairing is the idea you can use fastest.