New Intel Processor and 192 GB/256 GB RAM
-
For clarification: it’s for a proxmox instance. I wanna use the ram for open webzine/ollama.
-
If you're really going to need that much RAM, start looking at servers with multiple sockets. They support absurd amounts of RAM in a single chassis. I think the biggest regularly-available servers have four sockets, but all but the most basic have two.
-
What the heck are you self-hosting that anything beyond 64G is even taken into account?
-
I personally believe you are overbuilding. For example my OpenMediaVault Samba Server and DLNA server runs on a SingleBoard that has 256 megabytes of RAM. Yes MB. And it still has RAM free without swap.
-
I'd say this is the correct answer. If you're actually using that much RAM, you probably want it connected to the processor with a wide (fast) bus. I rarely see people do it with desktop or gaming processors. It might be useful for some edge-cases, but usually you want an Epyc processor or something like that, or it's way too much RAM.
-
What is openwebzine? Can't find any info on it.
-
Look up what system vendors will sell for that CPU. If they sell 256 GiB, then you are likely good.
I don't find I ever upgrade after the first couple months. I would max it out or get multi CPU boards wherI cannot afford to max it out.
-
And 4 sticks ate 4 times more prone to break down.
-
I'e seen that some want it to host their own LLM. It's far cheaper to buy DDR5 memory than somehow getting 100+ GB of VRAM. Whether or not this is a good idea is another question
-
sorry, fat fingers on tablet: I mean "open webui".
-
Oh, I'm not using it for OMV and Samba. I'm using it for ollama/open webui with RAM instead of VRAM.
-
My edge case is: I wanna spin up an ai-lxc in proxmox. ollama and open webui. using RAM instead of vram. but it should low on power consumption on idle. thats why I want an intel i-9 oder core ultra 9 with maxed out RAM. it idles on low power, but can run bigger ai-models using RAM instead of VRAM. it would be not so fast like with GPUs, but thats OK.
-
AI inference is memory-bound. So, memory bus width is the main bottleneck. I also do AI on an (old) CPU, but the CPU itself is mainly idle and waiting for the memory. I'd say it'll likely be very slow, like waiting 10 minutes for a longer answer. I believe all the AI people use Apple silicon because of the unified memory and it's bus width. Or some CPU with several memory lanes.
-
All current popular AI is meant to run on GPU. Why are you going to spend more money to run it on hardware for which it isn't intended?
-
Twice, because usually it's two sticks.
In any case, RAM failure is rare enough that quadrupling its chances is not gonna make any meaningful difference. Even if it does, RAM is the easiest thing to replace in a PC. Don't even need to go offline while waiting for a new stick. Someone who's got the cash to build that thing in the first place won't be too upset by the cost of another 32gb stick either, I don't think.
-
This isn’t really true — a lot of the newer MoE models run just fine on a CPU coupled with gobs of RAM. Yes, they won’t be quite as fast as a GPU, but getting 128GB+ of VRAM is out of reach of most people.
You can even run Deepseek R1 671b (Q8) on a Xeon or Epyc with 768GB+ of RAM, at 4-8 tokens/sec depending on configuration. A system supporting this would be at least an order of magnitude cheaper than a GPU setup to run the same thing.
-
If you need that much ram, I would look at getting a AMD Threadripper. If you need DDR5, the 7960X is their “budget” option, or if you are fine with DDR4, there is lots of used 3000 and 5000 series for cheap.
-
Well, anecdotal evidence of course but with the exception where a PSU blew up (and damaged a whole lot of things) I only ever have had RAM stick problems since like -95. Three times. Over some 30-40 PCs.
-
So if I had more memory channels it would be better to have say ollama use the cpu versus the gpu?
-
Because there's no advantage to having this much RAM in an economy build. If you're looking to max out your mainboard RAM then you're looking for a thread ripper anyways, not some economy i9...
