Scrapers are eating git.kernel.org: bots load the servers harder than live humans

Konstantin Ryabitsev, who is responsible for the kernel.org infrastructure, has described the scale of a problem he calls the “background radiation” of pesky crawlers. The way he puts it, it all sounds almost routine, but the figures are striking: rendering commit pages for scrapers eats up more CPU time on the servers of the Linux kernel's official Git repository than any other kind of legitimate access — including ordinary git clone.

Fourteen cores down the drain

According to Ryabitsev, at any given moment fourteen CPU cores across the five geographically distributed git.kernel.org nodes are busy doing one single thing — turning git commits into HTML for bots. In other words, a significant share of the infrastructure paid for and configured for the sake of Linux kernel development is essentially serving automated visitors who crawl through the repository's history page by page.

The problem here lies precisely in the nature of a web interface to Git. Every commit is a separate page generated on the fly, and there are millions of such pages in the kernel's history. For a human it is convenient navigation; for a crawler it is an endless field to traverse, every step of which costs the host real computation.

Not just the kernel

The news caught the attention of Simon Willison, who looks at it from his own vantage point. He is the author of Datasette, a tool that publishes an enormous number of crawlable pages. And he admits the topic worries him more and more: any project that serves up a lot of machine-generated pages ends up in the same trap as kernel.org.

Behind the laconic “this troubles me” hides a very concrete fear. The wave of aggressive scraping — fueled in large part by the race for data to train language models — is turning the generous openness of public resources into a line item of expenses. Giving your data away to the world for free is pleasant right up until the moment you have to pay for that pleasure with CPU cores that could have been working for the benefit of the project itself.

The note offers no concrete remedies — it is more of an alarm signal from people who see the load with their own eyes. But the very fact that bots now load the Linux kernel infrastructure harder than live developers speaks to the new nature of the problem far more eloquently than any declarations.

Source: Simon Willison