Do any of you have a buttload of RAM sitting around?
-
Hi,
I have a friend who is looking to run a few simulations he has implemented in python and needs around 256GB of ram. He is estimating it will take a couple of hours, but he is studying economics so take that with a grain of salt
For this instance, I recommended GCP, but I felt a bit dirty doing that. So, I was wondering if any of you have a buttload of memory he can burrow? Generally, would you lend your RAM for a short amount of time to a stranger over the internet? (assuming internet acccess is limited to a signle ssh port, other necessary safeguards are in place)
i have 8 gigs thats been living on my desk for the last 4 years
-
First, define what you are asking for.
Do you want someone to send you a cardboard box full of RAM? Then forget it. Nobody would be stupid enough to lend that much expensive hardware to someone on the internet.
Or are you asking for someone to let you run random code on their PC for a few hours? Then forget it. Nobody would be stupid enough to open "a single SSH port" to someone on the internet to run potential malware on their PC.
That's exactly what cloud platforms are there for, and if you don't like google, get any other cloud provider.
Seconded. If they can't optimize their code (which, I have never seen applications require 256 gigs of ram even in FAANG so I find that doubtful), then they need to rent a machine. The cloud is where you rent it. If not Google, then AWS, Azure, Digital Ocean, any number of places let you rent compute
-
Needing that much RAM is usually a red flag that the algo is not optimized.
*looking at the 14TB cluster I had running for 18 hours
Yep, nobody would ever need that much memory
-
Hi,
I have a friend who is looking to run a few simulations he has implemented in python and needs around 256GB of ram. He is estimating it will take a couple of hours, but he is studying economics so take that with a grain of salt
For this instance, I recommended GCP, but I felt a bit dirty doing that. So, I was wondering if any of you have a buttload of memory he can burrow? Generally, would you lend your RAM for a short amount of time to a stranger over the internet? (assuming internet acccess is limited to a signle ssh port, other necessary safeguards are in place)
Tell your friend to open source the algorithm.. Somebody will surely point at a easy optimization.
100 others will just shit on your friend -
Seconded. If they can't optimize their code (which, I have never seen applications require 256 gigs of ram even in FAANG so I find that doubtful), then they need to rent a machine. The cloud is where you rent it. If not Google, then AWS, Azure, Digital Ocean, any number of places let you rent compute
Yeah, it's an economics student running something on python. I can guarantee that it's horribly unoptimized.
-
*looking at the 14TB cluster I had running for 18 hours
Yep, nobody would ever need that much memory
Wow, yea I think you win that contest lol.
-
Wow, yea I think you win that contest lol.
To be honest, it was a very paralell process. I could do a fraction of the compute, needing a fraction of the RAM, but taking a shit ton more time.
Also, theres no perfect machine for this use. I can have 3.5 times more RAM than needed, or start swapping and waste time.
-
Needing that much RAM is usually a red flag that the algo is not optimized.
Nope. Some algorithms are fastest when a whole data set is held into memory. You could design it to page data in from disk as needed, but it would be slower.
OpenTripPlanner as an example will hold the entire road network of the US in memory for example for fast driving directions, and it uses the amount of RAM in that ballpark.
-
Nope. Some algorithms are fastest when a whole data set is held into memory. You could design it to page data in from disk as needed, but it would be slower.
OpenTripPlanner as an example will hold the entire road network of the US in memory for example for fast driving directions, and it uses the amount of RAM in that ballpark.
Sure, that is why I said usually. The fact that 2 people replied with the same OpenStreetMap data set is kinda proving my point.
Also, do you need the entire US road system in memory if you are going somewhere 10 minutes away? Seems inefficient, but I am not an expert here. I guess it is one giant graph, if you slice it up, suddenly there are a bunch of loose ends that break the navigation.
-
Hi,
I have a friend who is looking to run a few simulations he has implemented in python and needs around 256GB of ram. He is estimating it will take a couple of hours, but he is studying economics so take that with a grain of salt
For this instance, I recommended GCP, but I felt a bit dirty doing that. So, I was wondering if any of you have a buttload of memory he can burrow? Generally, would you lend your RAM for a short amount of time to a stranger over the internet? (assuming internet acccess is limited to a signle ssh port, other necessary safeguards are in place)
AWS has an r4.8xlarge 244gb ram with 32 vcores for $2.13 an hour If they can handle Linux. $2.81 an hour for windows.
-
Hi,
I have a friend who is looking to run a few simulations he has implemented in python and needs around 256GB of ram. He is estimating it will take a couple of hours, but he is studying economics so take that with a grain of salt
For this instance, I recommended GCP, but I felt a bit dirty doing that. So, I was wondering if any of you have a buttload of memory he can burrow? Generally, would you lend your RAM for a short amount of time to a stranger over the internet? (assuming internet acccess is limited to a signle ssh port, other necessary safeguards are in place)
Apply for compute time at a university cluster. It is free and usually easy.
-
Sure, that is why I said usually. The fact that 2 people replied with the same OpenStreetMap data set is kinda proving my point.
Also, do you need the entire US road system in memory if you are going somewhere 10 minutes away? Seems inefficient, but I am not an expert here. I guess it is one giant graph, if you slice it up, suddenly there are a bunch of loose ends that break the navigation.
I host routing for customers across the US, so yes I need it all. There are ways to solve the problem with less memory but the point is that some problems really do require a huge amount of memory because of data scale and performance requirements.
-
Seconded. If they can't optimize their code (which, I have never seen applications require 256 gigs of ram even in FAANG so I find that doubtful), then they need to rent a machine. The cloud is where you rent it. If not Google, then AWS, Azure, Digital Ocean, any number of places let you rent compute
In computational physics, 256g of RAM can be small. I have ran things that big easily. Some colleges have needed TiB. If it is heavy matrix diagonalization it can get chunky of unoptimized.
