AI crawlers cause Wikimedia Commons bandwidth demands to surge 50%.
-
To have the most recent data?
To just have the most recent data within reasonable time frame is one thing. AI companies are like "I must have every single article within 5 minutes they get updated, or I'll throw my pacifier out of the pram". No regard for the considerations of the source sites.
-
They can also crawl this publically-accessible social media source for their data sets.
Crawling would be silly. They can simply setup a lemmy node and subscribe to every other server. Activitypub crawler would be much more efficient as they wouldn't accidentally crawl things that haven't changed, but instead can read the activitypub updates.
Sure but we're in the comments section of an article about wikipedia being crawled, which is silly because they could just download a snapshot of wikipedia
-
Doesn't make any sense. Why would you crawl wikipedia when you can just download a dump as a torrent ?
Apparently the dump doesn't include media, though there's ongoing discussion within wikimedia about changing that. It also seems likely to me that AI scrapers don't care about externalizing costs onto others if it might mean a competitive advantage.
-
This is a cool use case. Just make sure you retain your own voice! If you read an AI-generated sentence out loud and think "I'd have said it this way instead", you should absolutely then change it to be that way.
Understood and I do. I try to tweak it a little to my own style. But it helps write the hundreds of cover letters I’m submitting a day. Looking for work. This usually took me hours for just one submission. Now I can fly through.
-
Doesn't make any sense. Why would you crawl wikipedia when you can just download a dump as a torrent ?
There's a chance this isn't being done by someone who only wants Wikipedia's data. As the amount of websites you scrape increases, your desire to use the easy tools loses out to creating the most general tool that can look at most webpages.
-
Nepenthes does about the same thing but isn't managed by a corp.
There's also Anubis, but it uses proof of work not a maze.
-
what assholes .. just fucking download the full package and quit hitting the URL
Scraper bots don't read instructions, they just follow links
-
Sure but we're in the comments section of an article about wikipedia being crawled, which is silly because they could just download a snapshot of wikipedia
That's right. It's not humans making careful decisions about what to download. It's a program that follows links and saves files.
-
Yes, but neither of those write as cleanly. And both are still prone to fragmenting, even if the fragments aren't conductive.
Charcoal is more dusty and more conductive than pencil "lead", which is pretty much processed charcoal and glue.
-
System shared this topic
