There’s one line in Anthropic’s new case study that sticks. Silas Alberti, SVP of Research at Cognition, describes handing the model a task before bed: “Okay, just please keep working on this and don’t stop until I wake up.” By morning, Fable 5 had worked eight hours straight — and had actually made real progress. “I hadn’t seen that before.”
A team that trusts no eval
Cognition has been building Devin, an autonomous AI software engineer, since early 2024. Its customers range from startups to Fortune 500 companies, so the bar is high: Devin’s code has to be production-ready. A bug slipped in quietly can get expensive downstream.
Alberti’s team has tested nearly every Claude generation — which is exactly why they’re hard to impress. “We’ve been burned like this a bunch of times”: models that ace a benchmark and then fall apart in real use. So their principle is blunt: “We trust no eval.” Instead, their best developers put each new model through a real day of work. The only question that matters: would you actually keep this code?
Where earlier models hit a wall
One limit held for a long time: how long can an agent run before it loses the thread? “Before Fable, you could delegate agents that could stay on-task for a couple of minutes, maybe an hour.” After that, sessions drifted. Give an earlier model five ideas at once and it would get tangled. On one database migration, an older Opus technically finished the job — but introduced a string of subtle bugs along the way.
Fable 5 clears their own bar
Cognition grades models on its own benchmark, Frontier Code — Alberti calls it an “anti-slop” standard, because existing evals too often rewarded code that passes tests but wouldn’t survive a real codebase. On the hardest subset, the earlier Opus scored around 10 percent. Fable 5 hit about 30.
The team’s first reaction was suspicion. “Is there a bug? This can’t be true.” But this time the dogfooding matched the numbers. Fable 5 stayed clear-headed in messy context, was the first model to properly use Cognition’s internal debugging tools, and — during triage — actually stated what it didn’t know, which Alberti says is what rebuilds trust.
My take
I’m always cautious with vendor case studies — of course Anthropic is selling its own model here. But the Cognition angle is interesting because it comes from a team that openly says it trusts no benchmark. The real jump isn’t a higher score, it’s the time horizon: from “a couple of minutes” to “eight hours overnight.”
That’s where it gets interesting for all of us. Alberti expects that in a year or two, 90 percent of agent sessions will be proactive ones — the agent finds the problem itself, scans the codebase, and messages you with the fix. That’s a different working model than “prompt in, answer out.” And it’s closer than many think.
Sources: