Cloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.
-
and try to slam your site with like 200+ requests per second
Your solution would do nothing to stop the crawlers that are operating 10ish rps. There's ones out there operating at a mere 2rps but when multiple companies are doing it at the same time 24x7x365 it adds up.
Some incredibly talented people have been battling this since last year and your solution has been tried multiple times. It's not effective in all instances and can require a LOT of manual intervention and SysAdmin time.
https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/
It's worked alright for me. Your mileage may vary.
If someone is scraping my site at a low crawl rate I honestly don't care so long as it doesn't impact my performance for everyone else. If I hosted anything that was not just public knowledge or copy regurgitated verbatim from the bumf provided by the vendors of the brands I sell, I might oppose to it ideologically. But I don't. So I don't.
If parallel crawling from multiple organizations legitimately becomes a concern for us I will have to get more creative. But thus far it hasn't, and honestly just wholesale blocking Amazon from our shit instantly solved 90% of the problem.
-
the only problem with that solution being applied to generic websites is schools and institutions can have many legitimate users from one IP address and many sites don't want a chance to accidentally block one.
This is fair in those applications. I only run an ecommerce web site, though, so that doesn't come into play.
-
How can authority not exist? That's staggeringly broad
given what domains we're hosted on; i think we've both had a version of this conversation about a thousand times, and both ended up where we ended up. do you want us to explain hypothetically-at-but-mostly-past each other again? I can do it while un-sober, if you like.
-
everyone remembers tomogatchi, they were like a digital houseplant.
-
Ok, I now need a screensaver that I can tie to a cloudflare instance that visualizes the generated "maze" and a bot's attempts to get out.
You probably just should let an AI generate that.
-
This post did not contain any content.
Will it actually allow ordinary users to browse normally, though? Their other stuff breaks in minority browsers. Have they tested this well enough so that it won't? (I'd bet not.)
-
This post did not contain any content.
Cloudflare kind of real for this. I love it.
It makes perfect sense for them as a business, infinite automated traffic equals infinite costs and lower server stability, but at the same time how often do giant tech companies do things that make sense these days?
-
Next step is an AI that detects AI labyrinth.
It gets trained on labyrinths generated by another AI.
So you have an AI generating labyrinths to train an AI to detect labyrinths which are generated by another AI so that your original AI crawler doesn't get lost.
It's gonna be AI all the way down.
All the while each AI costs more power than a million human beings to run, and the world burns down around us.
-
The problem I see with poisoning the data is the AI's being trained for law enforcement hallucinating false facts used to arrest and convict people.
They aren't poisoning the data with disinformation.
They're poisoning it with accurate, but irrelevant information.
For example, if a bot is crawling sites relating to computer programming, or weather, this tool might lure the crawler into pages related to animal facts, or human biology.
-
Next step is an AI that detects AI labyrinth.
It gets trained on labyrinths generated by another AI.
So you have an AI generating labyrinths to train an AI to detect labyrinths which are generated by another AI so that your original AI crawler doesn't get lost.
It's gonna be AI all the way down.
LLMs tend to be really bad at detecting AI generated content. I can’t imagine specialized models are much better. For the crawler, it’s also exponentially more expensive.
-
So the world is now wasting energy and resources to generate AI content in order to combat AI crawlers, by making them waste more energy and resources. Great!
The energy cost of inference is overstated. Small models, or “sparse” models like Deepseek are not that expensive to run. Training is a one-time cost that still pales in comparison to industrial processes.
Basically, only Altman wants it to be cost prohibitive so he can have a monopoly. Also, he’s full of shit.
-
This post did not contain any content.
So the web is a corporate war zone now and you can choose feudal protection or being attacked from all sides. What a time to be alive.
-
the used market
-
I would love to think so. But the word "verified" suggests more.
IP verification is a not uncommon method for commercial crawlers
-
given what domains we're hosted on; i think we've both had a version of this conversation about a thousand times, and both ended up where we ended up. do you want us to explain hypothetically-at-but-mostly-past each other again? I can do it while un-sober, if you like.
Not who you responded to but yeah I want to hear a drug fuelled rant I don't even care what topic
-
Any accessibility service will also see the "hidden links", and while a blind person with a screen reader will notice if they wonder off into generated pages, it will waste their time too.
Also, I don't know about you, but I absolutely have a use for crawling X, Google maps, Reddit, YouTube, and getting information from there without interacting with the service myself.
I'd assume they're using aria tags to hide the links from screen readers, at least that's what the article seems to imply.
-
This post did not contain any content.
And this, ladies and gentleman, is how you actually make profits on AI.
-
This post did not contain any content.
Spiderman pointing at Spiderman meme.
-
Next step is an AI that detects AI labyrinth.
It gets trained on labyrinths generated by another AI.
So you have an AI generating labyrinths to train an AI to detect labyrinths which are generated by another AI so that your original AI crawler doesn't get lost.
It's gonna be AI all the way down.
Yeah that’s how we achieve singularity.
-
This post did not contain any content.
Now this is a AI trap worth using. Don't waste your money and resources hosting something yourself, let Cloudflare do it for you if you don't want AI scraping your shit.
