Anthropic published a field report from its own engine room yesterday, and it is far more concrete than most blog posts about AI and software development. Sachin Malhotra describes how the service that decides which tests run on which pull request nearly fell over three times.
The numbers first
Anthropic engineers ship on average eight times as much code per quarter as they did between 2021 and 2025. Claude writes 80% of it and does a large share of the reviewing and approving too. The number of tests in the codebase grew tenfold; headcount barely moved. Result: a 25x increase in CI jobs over six months.
Malhotra’s line about it: writing code is no longer the constraint. Once review speeds up as well, CI is what starts to hurt.
What the service actually does
Running every test on every change works up to a certain size. After that, CI gates get long, expensive and untrustworthy. So Anthropic selects: a «listener» records the test results from every CI run, and a «selector» reads that history to decide which tests run on a new PR.
A detail worth stealing: humans are pretty good at spotting which failing test has nothing to do with them. Agents need context for that. Hand them a clean, relevant set of tests instead and they self-verify and keep moving.
The catch was that the per-test history lived in one process, so there was exactly one writer, and it could not be split. With several CI jobs per second, the listener fell behind. Twenty minutes of lag meant tens of thousands of test updates never reaching the selector.
70 days, 29 days, less than one
The pager started in October. Fix one: double the cores. It held for 70 days.
Along the way Malhotra kept a long-running session open in an internal version of Claude Tag, watching the service. Whenever the listener lag passed 50,000 jobs, Claude pinged him and picked the conversation back up where it had stopped. Claude argued for an overhaul for months. They chose the next patch for months.
Fix two, in February: one shard per package instead of a single writer for everything. It held 29 days. Fix three, in March: daily restarts. That one held less than a day, and the service kept sliding further behind while it did.
The rebuild
Then they took Claude’s advice: an in-memory data store. Any listener worker appends its result to a journal and moves on, holding nothing in memory. A separate consumer rolls the journal into per-test history every few seconds. Stateless, so horizontally scalable. More expensive to run, but you can measure it.
Three weeks, one engineer. A year ago that would have been closer to a quarter. The backlog has been flat since.
Plan for 25x
Malhotra’s advice closes the piece, and it is why I’m linking it here: assume your architecture will be at 25x load within two quarters. «Over-engineering» is losing some of its sting as an accusation, because the bar keeps moving. Designing your v0 for ten to twenty times the load you can see is fine, budget permitting.
The part that stays with me is different. The architecture was not really the mistake. The mistake was picking the cheap option three times while the assistant sitting next to them kept proposing the rebuild. Rebuilding now costs a fraction of what it used to. Patching still costs the same amount of attention it always did.
Sources: