How to combat large amounts of Ai scrapers
-
In my case I use https://www.bunkerweb.io/ as my proxy for that, but there are other tools like for example https://github.com/TecharoHQ/anubis
bunkerweb looks intresting
-
You probably don't need me to tell you, but keep good backups. Friend of mine recently had his account nuked without any reason given, and without the possibility of recourse.
as I heard that's pretty common at oracle, but it's good to spread the word
-
I understand, but the shift in user behaviour is significant and I think websites are not taking it into account. If the users move more and more to AI, and since Google introduced AI mode it's only a question of time until it becomes the default, we will see more and more of what we thing are AI crawlers and less and less organic users.
AI seems to be the new middleman between you and the user, and if you block the middleman, you block the user. For people with hobby websites or established sites it may make sense because people either know of them, or getting more exposure is not a wish or requirement, but for everyone else, it will be painful.
So, what I'm reading is, if your "users" are bad (or bots), just get better users.
Sounds like a net win.
-
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
wrote last edited by [email protected]What's bothering you?
- Is it to give out data for AI training? I guess you can't fundamentally protect against this, except by limiting how much content is provided to each address.
- Or is it the resource strain that it causes on your server? In that case i recommend limiting how much a single client / IP address can request in a day.
-
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
does anubis not work?
-
does anubis not work?
i can only get it to protect one container. i have 3 that i need protected and i cant figure out how to run more then one instance of it.
-
What's bothering you?
- Is it to give out data for AI training? I guess you can't fundamentally protect against this, except by limiting how much content is provided to each address.
- Or is it the resource strain that it causes on your server? In that case i recommend limiting how much a single client / IP address can request in a day.
its the strain of it i mostly run instances and frontends so the training is not a huge problem
-
If nginx, here's an open-source blocker/honeypot: https://github.com/raminf/RoboNope-nginx
If you have it set up to be proxied or hosted by Cloudflare, they have their own solution: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/
ill check robonope out seems promising
-
Anubis is the name of the tool. Also, Cloudflare just announced they have something against AI scrapers.
ive been using Anubis my only issue is i would have to run more then one instance and i dont like cloudflare personaly
-
its the strain of it i mostly run instances and frontends so the training is not a huge problem
wrote last edited by [email protected]the keyword you need is "DDoS protection" i guess
it keeps the server from getting overloaded due to too many requests
