Who Does Anubis Actually Stop?

Anubis is a program that many open source projects use to protect their sites and repositories from aggressive automated crawlers, including bots that harvest data to train AI models. The tool requires visitors to pass a computational challenge (proof-of-work) before they can access a page — a mechanism similar to those used to guard against spam and DDoS attacks.

Over the past few months, Anubis has gained wide adoption among administrators of open source infrastructure who are tired of the load created by unscrupulous scrapers.

In their post, the blog's author asks just how effective this protection really is. The question is who exactly the proof-of-work barrier stops: primitive bots that crawl sites en masse without regard for their policies, or more technically capable operators who can bypass such restrictions without much effort.

Discussions like this reflect a broader problem facing the open source developer community: the growing infrastructure load from companies collecting data to train language models. Solutions like Anubis have become an attempt to answer this problem through technical means, but the question of how proportionate and effective such measures are remains a subject of debate in the community.

The post sparked a discussion among readers of the Lobsters aggregator, where various points of view were weighed on the balance between protecting infrastructure and keeping things convenient for ordinary users.

Source: Lobsters