Konstantin Ryabitsev runs the infrastructure behind git.kernel.org, the official home of the Linux kernel repositories. On September 7 he put a number into circulation that traveled fast among developers.
The math
Across five geographically distributed servers, 14 CPU cores are permanently busy rendering Git commits as HTML pages — for scrapers. In his words: they spend more CPU cycles rendering commits for scrapers than they spend on all other kinds of legitimate access, including git clones.
That’s the part that stings. Not that crawlers create load; they always have. It’s that bot load has overtaken load from the people the service exists for. A git clone is cheap by comparison: one request, one packfile, done. A crawler instead walks every single commit as its own HTML page, and the server has to build each of those from scratch.
Why this matters to anyone hosting anything
Simon Willison, who picked the story up, is thinking about his own project. Datasette builds pages dynamically out of databases. Every filter combination is its own URL, and every one of them looks to a crawler like another page worth grabbing. The same goes for wikis, issue trackers, map services, archives — anything with more URLs than content.
The reason I think this story matters: it makes a bill visible that normally shows up nowhere. Training data is a raw material on one side and a server invoice on the other. At a sponsored project like kernel.org, someone notices. For the person running an archive on a small box in the evening, it just quietly eats the budget.
robots.txt helps exactly as long as someone chooses to honor it. What actually helps is less pleasant: rate limits on expensive views, static exports instead of on-demand rendering, and a willingness to block whole paths for anything that doesn’t show up with a browser. That isn’t a fix. It’s damage control — and right now it’s the only thing that works.
Sources: Simon Willison: Creepy crawlies