Internet Archive played crucial role in tracking shady CDC data removals
-
Is there a way to distribute it so everyone just has parts of it? Aren't there p2p cloud storage solutions that exist?
-
Here's how to help them: https://github.com/ArchiveTeam/warrior-dockerfile
-
A couple years ago I read that Filecoin has teamed up with the internet archive to synchronize the data on the Blockchain. I'm not sure how far they are yet, but it's something that could work if it doesn't turn out to be just crypto hype in the end.
-
although could have been avoidable, it begs the question who was behind the attacks.
I think we can safely say it was Peelon Shmusk, the worlds worst spy!
-
When the internet archive was attacked a few months ago we were like "who would be dumb and mean enough to do that?". We have new suspects!
-
Oh, cool, didn't know about this, throwing it on my home lab now.
-
The problem is you'd need to split it down to an amount that people would be happy hosting and then host it multiple times in case any node goes offline.
Another comment in the thread says it's likely over 100PB today (100,000 terabytes). I'd say 4 copies (spread over different time zones) is a relatively minimal level of redundancy (people may host on machines that aren't powered all the time), and you'd get a network with the most participants at around the 150gb per node mark.
That comes to nearly 3 million participants needed
Which isn't insurmountable, but also not remotely easy to get going from nothing
-
That's not the Internet Archive; that's a separate group (ArchiveTeam). They're completely unrelated. They use the Internet Archive for storage but are otherwise completely unrelated. The data archived by Archive Team Warrior does not go into the Wayback Machine.
-
This comment from 8 months ago says 152PB: https://www.reddit.com/r/DataHoarder/comments/1cu79ke/the_archiveteam_has_a_cost_shameboard_of_the_top/l4om4m6/
-
These guys seem cool but they're not the archive.org from the op article
-
As I understand it, their data does in fact enter into the Wayback Machine. They are just also available in the direct WARC archive files(which IMO sounds beneficial to the idea of exporting in bulk to another backup host). At least that’s how their FAQ reads.
And given that they focus on web crawling, and not other arbitrary data formats that IA accepts, 2.8% of over 100 petabytes is still a respectable amount of data.
That said, help is help. If another archival project team wants me to run a worker node so they can distribute load and dodge crawler blocks, let me know, I’ve got space.
-
It's a team of volunteers who help scrape and upload things to archive.org.
-
It does go into the WaybackMachine AFAIK.
-
Need an archive of the archive
-
It doesn't help that people put silly things onto the IA. I've seen some things like YouTube videos that really didn't need to be there (they have, objectively, nothing of value enough to warrant taking up space on these servers that could be used for more important materials..).
-
I literally posted a comment saying "sure is odd that this is happening right before the election. Not saying it means anything, but maybe it's not a coincidence?" and got downvoted to hell lmao.
-
If they added download options on different taxonomies, I'd try to grab some things to archive.
-
Yeah, stuff that is able to be taught is vital to have archives, but some Twitch streamer playing some MMO/shooter/scary game isn't what I would consider very imperative to get backed up.
-
As I understand it, their data does in fact enter into the Wayback Machine
Thanks for the info! It never used to, so I guess that changed at some point.
