Bots are eating git.kernel.org: only 2% of requests come from humans

Konstantin Ryabitsev, who is responsible for the kernel.org infrastructure, has published a note with concrete figures on what the invasion of AI crawlers is costing the community. The git.kernel.org server, which hosts the Linux kernel repositories, takes in about 6 million requests a day, each of them demanding to be shown some random commit.

A defense they've learned to bypass

To filter out automated traffic, the site runs the Anubis mechanism — it forces the client to solve a computational challenge before letting it through. Two thirds of the requests (66%) are turned away by this barrier right off the bat. But the remaining third dutifully solves the math puzzle and breaks through to the main site anyway. Ryabitsev jokes that the contents of the kernel repositories must be so valuable that it's worth burning a heap of CPU cycles on solving the Anubis challenge for their sake.

Who this really is

There's no way to tell a bot from a live human with complete certainty. But logic suggests that if someone is requesting an old commit in some random, long-abandoned fork, it's hardly a real developer going about their work. Adding together a whole pile of generous assumptions in favor of humans, Ryabitsev arrives at a sobering conclusion: legitimate requests make up only about 2% of all git.kernel.org traffic. The rest are scrapers.

The paradox of the situation was noted in the discussion of the note as well. It would be far more efficient to clone the repository once and then pull in updates from time to time — instead of continuously hammering the web interface and effectively causing an overload. There's also a ready-made mirror on GitHub that can be scraped without harming the community's infrastructure. But the operators of these training pipelines apparently prefer brute force.

The problem isn't unique to the Linux kernel. Similar complaints were recently voiced by the Wikimedia Foundation: the organization deliberately made its entire knowledge base available for a one-click download, yet the language models keep pounding on the web servers as if it were 1995. Behind all of this lies the same ailment — indifference to the common good: resources maintained for years by volunteers and sponsors are treated as free fodder, whose cost the consumer never stops to consider.

Source: LWN.net