Anubis - Weighs the soul of incoming HTTP requests using proof-of-work to stop AI crawlers
-
I not find any instruction on the source page on how to actually deploy this. That would be a nice touch imho.
Or even a quick link to the relevant portion of the docs at least would be cool
-
generates an infinitely deep tree
Wouldn't the bot simply limit the depth of it's seek?
It could be infinitely wide too if they desired. It shouldn't be that hard to do I wouldn't think. I would suspect they limit the time a chain can use though to eventually escape out, though this still protects data because it obfuscates legitimate data that it wants. The goal isn't to trap them forever. It's to keep them from getting anything useful.
-
Why Sha256? Literally every processor has a crypto accelerator and will easily pass. And datacenter servers have beefy server CPUs. This is only effective against no-JS scrapers.
It requires a bunch of browser features that non-user browsers don't have, and the proof-of-work part is like the least relevant piece in this that only gets invoked once a week or so to generate a unique cookie.
I sometimes have the feeling that as soon as some crypto-currency related features are mentioned people shut off part of their brain. Either because they hate crypto-currencies or because crypto-currency scammers have trained them to only look at some technical implementatiin details and fail to see the larger picture that they are being scammed.
-
It is isn't on many levels.
-
It only runs against the Firefox user agent. This is not great as the user agent can easy be changed. It may work now but tomorrow that could all change.
-
It doesn't measure load so even if your website has only a few people accessing it they will stick have to do the proof of work.
-
The POW algorithm is not well designed and requires a lot of compute on the server which means that it could be used as a denial of service attack vector. It also uses sha256 which isn't optimized for a proof of work type calculation and can be brute forced pretty easily with hardware.
In summary the Tor implementation is a lot better. I would love to see someone port it to the clearnet.
I use https://sx.catgirl.cloud/ so I'm already primed to have anime catgirls protecting my webs.
-
-
I use https://sx.catgirl.cloud/ so I'm already primed to have anime catgirls protecting my webs.
Catgirls, jackalgirls, all embarrassing. Go full-on furry.
-
For those not aware, nepenthese is an example for the above mentioned approach !
This looks like it can can actually fuck up some models, but the unnecessary CPU load it will generate means most websites won't use it unfortunately
-
What's the ffxiv reference here?
Anubis is from Egyptian mythology.
-
What's the ffxiv reference here?
Anubis is from Egyptian mythology.
The names of release versions are famous FFXIV Garleans
-
It's a clever solution but I did see one recently that IMO was more elegant for noscript users. I can't remember the name but it would create a dummy link that human users won't touch, but webcrawlers will naturally navigate into, but then generates an infinitely deep tree of super basic HTML to force bots into endlessly trawling a cheap-to-serve portion of your webserver instead of something heavier.
That's a tarpit that you're describing, like iocaine or nepthasis. Those are to feed the crawler junk data to try and make their eventual output bad.
Anubis tries to not let the AI crawlers in at all.
-
It requires a bunch of browser features that non-user browsers don't have, and the proof-of-work part is like the least relevant piece in this that only gets invoked once a week or so to generate a unique cookie.
I sometimes have the feeling that as soon as some crypto-currency related features are mentioned people shut off part of their brain. Either because they hate crypto-currencies or because crypto-currency scammers have trained them to only look at some technical implementatiin details and fail to see the larger picture that they are being scammed.
So if you try to access a website using this technology via terminal, what happens? The connection fails?
-
So if you try to access a website using this technology via terminal, what happens? The connection fails?
If your browser doesn't have a Mozilla user agent (I.e. like chrome or Firefox) it will pass directly. Most AI crawlers use these user agents to pretend to be human users
-
generates an infinitely deep tree
Wouldn't the bot simply limit the depth of it's seek?
That would be reasonable. The people running these things aren't reasonable. They ignore every established mechanism to communicate a lack of consent to their activity because they don't respect others' agency and want everything.
-
If your browser doesn't have a Mozilla user agent (I.e. like chrome or Firefox) it will pass directly. Most AI crawlers use these user agents to pretend to be human users
What I'm thinking about is more that in Linux, it's common to access URLs directly from the terminal for various purposes, instead of using a browser.
-
What I'm thinking about is more that in Linux, it's common to access URLs directly from the terminal for various purposes, instead of using a browser.
If you're talking about something like
curl, that also uses its own User agent unless asked to impersonate some other UA. If not, then maybe I can't help.
