New Intel Processor and 192 GB/256 GB RAM
-
This isn’t really true — a lot of the newer MoE models run just fine on a CPU coupled with gobs of RAM. Yes, they won’t be quite as fast as a GPU, but getting 128GB+ of VRAM is out of reach of most people.
You can even run Deepseek R1 671b (Q8) on a Xeon or Epyc with 768GB+ of RAM, at 4-8 tokens/sec depending on configuration. A system supporting this would be at least an order of magnitude cheaper than a GPU setup to run the same thing.
-
If you need that much ram, I would look at getting a AMD Threadripper. If you need DDR5, the 7960X is their “budget” option, or if you are fine with DDR4, there is lots of used 3000 and 5000 series for cheap.
-
Well, anecdotal evidence of course but with the exception where a PSU blew up (and damaged a whole lot of things) I only ever have had RAM stick problems since like -95. Three times. Over some 30-40 PCs.
-
So if I had more memory channels it would be better to have say ollama use the cpu versus the gpu?
-
Because there's no advantage to having this much RAM in an economy build. If you're looking to max out your mainboard RAM then you're looking for a thread ripper anyways, not some economy i9...
-
He goes over the different ways to run a selfhost AI without a GPU, like you want to do, including maxing RAM and using PCI-e M.2 add-on boards.
-
The i9-10900 has 4 channels (Quadro-Channel DDR4-2933 (PC4-23466, 93.9GB/s).
would this be better in this way than an i9-14xxx (Dual-Channel DDR5-5600 (PC5-44800, 89.6GB/s))?does the numbers (93 GB/s and 89GB/s) mean the speed for a RAM-stick or the speed all together? maybe an old i9-10xxx with 4channel-ram was better than a new dual-channel.
-
Thank you very much! This leads to this article: https://forum.level1techs.com/t/deepseek-deep-dive-r1-at-home/225826/2
Maybe the 9959x is what I am looking for. -
Well, the numbers I find on google are: a Nvidia 4090 can transfer 1008 GB/s. And a i9 does something like 90 GB/s. So you'd expect the CPU to be roughly 11 times slower than that GPU at fetching numbers from memory.
-
Seems to mean all together. (5600MT/s / 1000) x 2 sticks x 64bit / 8bits/Byte = 89.6 GB/s
or 2933/1000 x 4 x 64bit / 8 = 93.9 GB/s
so they calculated with double the DDR bus width, or 4times the bus width.
-
OP wants to store all of their porn collection in RAM
