If Anubis is so great, why isn't it everywhere?

[davriellelouna@lemmy.world]

Disclaimer: I don't have a background in computer science

I recently heard about a lightweight opensource software called Anubis.

Anubis is designed to stop AI crawlers that download a lot of data to train artificial intelligence models

https://anubis.techaro.lol/

Several websites have deployed Anubis:

https://gitlab.gnome.org/GNOME

10 websites have deployed. 10 websites. Out of millions of websites.

My question is extremely simple.

If this software is so damn great, why isn't it everywhere?

Seriously. Why isn't it used on Lemmy? On Wikipedia? On CBC?

[slazer2au@lemmy.world]

Because it's not a perfect solution and other sites have other solutions in place.

[fenrrs@lemmy.world]

seems like some crawlers already know how to bypass it: https://social.anoxinon.de/@Codeberg/115033782514845941

[sbv@sh.itjust.works]

Weird that this is getting downvotes. It's a legit question.

[sbv@sh.itjust.works]

Yeah. It's an arms race. Any technological defense will be countered eventually. In the long run, I'm not sure a technical defense like this will be sufficient - we'll need legal defenses that are enforced.

[argumentativemonotheist@lemmy.world]

People get sensitive when religion is mentioned!

[underline960@sh.itjust.works]

My guess is it's the tone/wording that implies that OP doesn't think it's great.

I agree that it's a legit question, though.

[kbal@fedia.io]

It isn't so great, and it is everywhere.

[renegadespork@lemmy.jelliefrontier.net]

Yeah, that’s the dilemma. The more popular it gets, the more crawlers will be designed to circumvent it.

[ephera@lemmy.ml]

I believe, (far too) much of the commercial world relies on Cloudflare to solve that problem.

And as for Wikipedia, any AI trainer worth their salt should know that they don't need to crawl it, because you can actually just download the whole Wikipedia dataset.

[stringere@sh.itjust.works]

Set is so much cooler.

[traches@sh.itjust.works]

• not every website has the problem it solves
• not everyone who does likes the solution it offers
• web development moves fast but not that fast

[zloubida@sh.itjust.works]

Pffff stop with your superstitions. Anybody serious knows that Teutates is the best.

[shadow@lemmy.ca]

It's not great, it's actually pretty ineffective.

https://lock.cmpxchg8b.com/anubis.html

[luffy879@lemmy.ml]

10 websites have deployed.

Source?

Because many already have solved the Problem via cloudflare, etc.

[icystar@lemmy.cif.su]

It's not that great, and the creator seems to have some weird fixation on little girls.

[icystar@lemmy.cif.su]

I think we're probably going to see a schism at some point in the fediverse.

The fragile people who can't seem to tolerate anything they don't like will end up in their own bubble.