How to combat large amounts of Ai scrapers
-
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
I've seen people suggesting and using Anubis, haven't used it myself though.
-
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
crowdsec is arguably not completely open source, but I'm very satisfied with it.
-
I've seen people suggesting and using Anubis, haven't used it myself though.
I was going to recommend that, very easy to setup
-
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
Wern't there a few AI maze projects in the works? I wonder if running one of those for a bit will cause you to be added to an ignore list, clearly they dont respect your robots file.
-
Wern't there a few AI maze projects in the works? I wonder if running one of those for a bit will cause you to be added to an ignore list, clearly they dont respect your robots file.
Tar pits I think is the term they use to pollute AI data.
-
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
You need yo block the alibaba subnets primarily. In my experience this is where most of them originate
-
I've seen people suggesting and using Anubis, haven't used it myself though.
i tried Anubis and it works great the only issue is it wont support multiple subdomains
-
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
I've seen people mention Anubis, the other one I heard about in a blog post that's maybe worth looking into is go-away.
-
crowdsec is arguably not completely open source, but I'm very satisfied with it.
The scraper blocklist on crowdsec requires a paid subscription, though, or did you find another workaround?
-
I've seen people suggesting and using Anubis, haven't used it myself though.
wrote last edited by [email protected]Second Anubis, just finished by setup yesterday i have it of a oracle cloud frre tier vps, which depending on the domain routes the traffic to services hosted on the vps itself or to my server ar home.
Relatively easy to setup, blocks most requests with very few false positives (one of which for example it would aggressively challenge by thunderbird trying to reach my baikal instance). I set a bit more aggresive rules than default (i also block googlebot and bingbot, since i received a bit more requests than I'd like). In like 10 hours it straight up denied about 5000 requests from the ai-catchall ruleset (mostly amazonbot) and challenged about 10000, mostly from a block of IPs in singapore, some of the hosts having the user agent of a Macintosh with PowerPC. They all sure love to explore the public repos on my git server.I'm in the process of changing servers for an upgrade, the old one still hosting more services while I setup the new one. The old one now does run audibly quiter. I don't even want to think how much electricity went wasted because of those bots
-
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
If you're able to, use GeoIP ranges to only allow access from the countries you want.
That immediately limits a lot of everything
Then - again if you're able to - use a block list that covers known scrapers in case they're in your country.
I use pfBlockerNG on my pfSense firewall for exactly this.
-
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
Anubis is the name of the tool. Also, Cloudflare just announced they have something against AI scrapers.
-
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
I just realized an interesting thing - if I use Gemini, and tell it to do deep research, it actually goes to the websites it knows/finds, and looks up the content to provide up-to-date answers. So, some of those AI crawlers are actually not crawlers, but actual users who just use AI instead of coming directly to the site.
Soo... blocking AI completely could also potentially reduce exposure, especially as more and more people use AI to basically do searches instead of browsing themselves. That would also explain the amount of requests daily - could be simply different users using AI to research for some topic.
Point is, you should evaluate if the AI requests are just proxies of real users, and blocking AI blocks real users from knowing your site exists.
-
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
Well, someone had great idea to use zipbombs. I saw it somewhere but I don’t remember where.
-
I just realized an interesting thing - if I use Gemini, and tell it to do deep research, it actually goes to the websites it knows/finds, and looks up the content to provide up-to-date answers. So, some of those AI crawlers are actually not crawlers, but actual users who just use AI instead of coming directly to the site.
Soo... blocking AI completely could also potentially reduce exposure, especially as more and more people use AI to basically do searches instead of browsing themselves. That would also explain the amount of requests daily - could be simply different users using AI to research for some topic.
Point is, you should evaluate if the AI requests are just proxies of real users, and blocking AI blocks real users from knowing your site exists.
some of those AI crawlers are actually not crawlers, but actual users who just use AI instead of coming directly to the site. Soo.. blocking AI completely could also potentially reduce exposure.
Normally, websites want users to come to their site, instead of an AI search engine "stealing" the content and presenting it as it's own. Yes, AI search engines are more convenient for the user, but in the end it will discourage website creators and thereby cut of it's own "food supply".
-
some of those AI crawlers are actually not crawlers, but actual users who just use AI instead of coming directly to the site. Soo.. blocking AI completely could also potentially reduce exposure.
Normally, websites want users to come to their site, instead of an AI search engine "stealing" the content and presenting it as it's own. Yes, AI search engines are more convenient for the user, but in the end it will discourage website creators and thereby cut of it's own "food supply".
I understand, but the shift in user behaviour is significant and I think websites are not taking it into account. If the users move more and more to AI, and since Google introduced AI mode it's only a question of time until it becomes the default, we will see more and more of what we thing are AI crawlers and less and less organic users.
AI seems to be the new middleman between you and the user, and if you block the middleman, you block the user. For people with hobby websites or established sites it may make sense because people either know of them, or getting more exposure is not a wish or requirement, but for everyone else, it will be painful.
-
The scraper blocklist on crowdsec requires a paid subscription, though, or did you find another workaround?
I don't remember how I set it up a long time ago. But when I look at my server logs I only see myself.
Afaik I just added the biggests lists. But I don't remember.
-
I've seen people suggesting and using Anubis, haven't used it myself though.
I’ve had trouble with it using a vpn and privacy browsers. It often blocks me until I use a default browser.
-
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
If nginx, here's an open-source blocker/honeypot: https://github.com/raminf/RoboNope-nginx
If you have it set up to be proxied or hosted by Cloudflare, they have their own solution: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/
-
some of those AI crawlers are actually not crawlers, but actual users who just use AI instead of coming directly to the site. Soo.. blocking AI completely could also potentially reduce exposure.
Normally, websites want users to come to their site, instead of an AI search engine "stealing" the content and presenting it as it's own. Yes, AI search engines are more convenient for the user, but in the end it will discourage website creators and thereby cut of it's own "food supply".
Yeah I'd consider blocking out both the bots and AI-users a win-win lmao
