Outgrown my Synology NAS, time for a proper dedicated machine
-
Great news! I started my selfhost journey over a year ago, and I'm finding myself needing better hardware. There's so many services I want that my NAS can't handle. And I unfortunately need to add GPU transcoding to my Jellyfin setup.
What's the best OS for a machine focused on containers and (getting started with) VMs? I've heard Proxmox
What CPU specs should I be concerned about?
I'm willing to buy a pre-built as long as its hardware has sufficient longevity.
-
Great news! I started my selfhost journey over a year ago, and I'm finding myself needing better hardware. There's so many services I want that my NAS can't handle. And I unfortunately need to add GPU transcoding to my Jellyfin setup.
What's the best OS for a machine focused on containers and (getting started with) VMs? I've heard Proxmox
What CPU specs should I be concerned about?
I'm willing to buy a pre-built as long as its hardware has sufficient longevity.
Honestly I just run Alpine Linux on a mini PC (router) or Raspberry Pi (NAS). I don’t like to screw around with outdated, bloated Debian-based distros.
-
Not at all. Proxmox does a great job at hosting VMs and giving a control plane for them - but it does not do containers well. LXCs are a thing, and it hosts those - but never try to do docker in an LXC. (I tried so many different ways and guides and there were just too many caveats, and you end up always essentially giving root access to your containers, so it's not great anyway). I'd like to see proxmox offer some sort of docker-first approach will it will manage volumes at the proxmox level, but they don't seem concerned with that, and honestly if you're doing that then you're nearing kubernetes anyway.
Which is what I ended up doing - k3s on proxmox VMs. Proxmox handles the instances themselves, spins up a VM on each host to run k3s, and then I run k3s from within there. Same paradigm as the major cloud providers. GKE, AKS, and EKS all run k8s within a VM on their existing compute stack, so this fits right in.
Thanks for your point of view ! I'm still new to proxmox and I went the LXC route... Seems to be working well so far but time will tell!
-
Not at all. Proxmox does a great job at hosting VMs and giving a control plane for them - but it does not do containers well. LXCs are a thing, and it hosts those - but never try to do docker in an LXC. (I tried so many different ways and guides and there were just too many caveats, and you end up always essentially giving root access to your containers, so it's not great anyway). I'd like to see proxmox offer some sort of docker-first approach will it will manage volumes at the proxmox level, but they don't seem concerned with that, and honestly if you're doing that then you're nearing kubernetes anyway.
Which is what I ended up doing - k3s on proxmox VMs. Proxmox handles the instances themselves, spins up a VM on each host to run k3s, and then I run k3s from within there. Same paradigm as the major cloud providers. GKE, AKS, and EKS all run k8s within a VM on their existing compute stack, so this fits right in.
Thanks for sharing your experience ! I was kinda interested for my new N300 if I should install promox+LXC-docker or promox+VM-docker !
Hearing you had a lot of issues and caveats makes my choice easier wihout even giving it a try ! So thanks !
-
Depending on how many bays your Synology is, you might be best off getting a nuc or a mini pc for compute and using your synology just for storage.
wrote last edited by [email protected]I did his when I moved from unraid because I wanted better infra as code for my dockers etc. Kept unraid with all my drives and use NFS mounts from another machine with proxmox that runs a VM for my dockers
-
Thanks for sharing your experience ! I was kinda interested for my new N300 if I should install promox+LXC-docker or promox+VM-docker !
Hearing you had a lot of issues and caveats makes my choice easier wihout even giving it a try ! So thanks !
I really wanted it to work, for me it made the most sense I thought, as little virtualization as I could do. VM felt like such a heavy layer in between - but it just wasn't meant to work that way. You have to essentially run your LXC as root, meaning that it's essentially just the host anyway so it can run docker. Then when you get down to it, you've lost all the benefits of the LXC vs just running docker. Not to mention that anytime there was even am minor update to proxmox something usually broke.
I'm surprised Proxmox hasn't added straight-up support for containers, either by docker, podman, or even just containerd directly. But, we aren't it's target audience either.
I'm glad you can take my years of struggling to find a way to get it to work well and learn from it.
-
My Synology is compatible with an expansion unit and can support two of them. Check if yours can do the same for the storage aspect.
Good looking out, I had no idea that was an option
-
Define goals. What services can’t be handled?
If transcoding is a goal build around intel. Quicksync video is a no brainer, imo. GPU is unnecessary power draw (15-25w+ idle depending on card) and waste of a pcie slot unless you want to do LLM stuff. Imo 10th gen intel is the sweet spot for quicksync unless you desperately need av1/vp9. If so then you need much more expensive 13/14 gen, which use more power and have more considerations for thermal management
OS is an endless debate. Proxmox is fine and free, why not try it? Unraid is easier to get your bearings but it does cost money. Debian is also free but a bit more confusing because not purpose built. Truenas as well. All can do containers and VMs, but approach in different ways. None is “best” but some are more “free” which is nice
CPU specs are dependent on goals. For transcoding as said above quicksync is necessary and is so impressive. I can transcode a 4k remux to one device while transcoding a 1080 remux to another and direct playing a 4k remux and cpu sits under 25% load on Xeon equivalent of 10700. You don’t need a Xeon btw, I just got a great deal where this was $50 (see next point). Otherwise specs depend wildly on what you plan to do. I can run windows VMs pretty well with this though for the handful of times I need a windows machine
Prebuilt is a waste. Used hardware is cheap and gives more options and can plan more. What are you willing to buy now and what do you eventually want? My NAS started as a 36tb array with 16gb ram and no cache, now it’s 234tb and 4tb cache with 32gb ecc ram years later. Slowly building up was easier on wallet and used hardware, refurb drives, etc is 100% the build. Your goals will likely vary but figure out your roadmap and go from there
Also keep in mind that not every service benefits from running on a NAS. My homeassistant server is run on a raspberry pi for example. Easier to keep it segregated and don’t have to worry about getting zwave/zigbee/mqtt/etc all working with a docker plus dealing with any server downtime impacting home. Tbf literally everything else is run on the nas though haha
In hindsight, I didn't explain myself well enough. My plan is to use my current NAS as a NAS and little more; I'd like a machine with respectable hardware to handle what my NAS is currently running plus more.
My NAS has Jellyfin, arrs, all the stuff that goes with that, Pi-hole, and Homarr. And that's pushing its limits: everything has been slow, streams freeze, I've had containers quit, etc.
I'd like to get into other projects like Radicale, Mealie, ErsatzTV (old PC could handle it, NAS can't), CCTV, and more. But according to my resources, the NAS can't handle it
GPU (for the sake of transcoding) isn't worth it?
-
Not at all. Proxmox does a great job at hosting VMs and giving a control plane for them - but it does not do containers well. LXCs are a thing, and it hosts those - but never try to do docker in an LXC. (I tried so many different ways and guides and there were just too many caveats, and you end up always essentially giving root access to your containers, so it's not great anyway). I'd like to see proxmox offer some sort of docker-first approach will it will manage volumes at the proxmox level, but they don't seem concerned with that, and honestly if you're doing that then you're nearing kubernetes anyway.
Which is what I ended up doing - k3s on proxmox VMs. Proxmox handles the instances themselves, spins up a VM on each host to run k3s, and then I run k3s from within there. Same paradigm as the major cloud providers. GKE, AKS, and EKS all run k8s within a VM on their existing compute stack, so this fits right in.
Docker runs fine nested in lxc with uid/gid mapping.
The difficulties of running docker in lxc are particular to proxmox, I ran docker in lxc on proxmox for years, but I'm glad I moved incus; much more sensible approach.
-
It's 4 bays, and we're eating that space up quicker than I imagined
I run a 4 bay and a N100 NUC.
The Synology is almost a pure storage machine. Works really well with Proxmox on the side. Not a single file has made it kneel yet, and I’ve thrown some high bitrate badboys on it.
Is not upgrading the drives an alternative?
I feel like you sacrifice a lot of practicality removing the NAS, such as automatic backup from phones and very easy remote access.
Personally I also prefer separating data and software, so I don’t lose it all if a component fails.Just my .02
-
I run a 4 bay and a N100 NUC.
The Synology is almost a pure storage machine. Works really well with Proxmox on the side. Not a single file has made it kneel yet, and I’ve thrown some high bitrate badboys on it.
Is not upgrading the drives an alternative?
I feel like you sacrifice a lot of practicality removing the NAS, such as automatic backup from phones and very easy remote access.
Personally I also prefer separating data and software, so I don’t lose it all if a component fails.Just my .02
That's my goal: use the NAS as a NAS, use a computer for containers and the like. I'm using Seagate Exos for the NAS exclusively
-
In hindsight, I didn't explain myself well enough. My plan is to use my current NAS as a NAS and little more; I'd like a machine with respectable hardware to handle what my NAS is currently running plus more.
My NAS has Jellyfin, arrs, all the stuff that goes with that, Pi-hole, and Homarr. And that's pushing its limits: everything has been slow, streams freeze, I've had containers quit, etc.
I'd like to get into other projects like Radicale, Mealie, ErsatzTV (old PC could handle it, NAS can't), CCTV, and more. But according to my resources, the NAS can't handle it
GPU (for the sake of transcoding) isn't worth it?
wrote last edited by [email protected]This makes sense
No sense in getting rid of hardware that is working. I’m not familiar with ersatztv but for all the other stuff I am able to handily run it on a 10th gen intel build that is also handling nas duties fwiw. And some stuff is not ideal (cctv is handled via blue iris, which runs in windows VM, everything else is docker)
for the gpu it really depends on your needs. How many users is the big one. If you have at most 2-4 concurrent users and that is an uncommon scenario the gpu is a waste of power, money, and thermal management. Igpu will sip power and transcode (depending on library content, again av1/vp9 on a 10th gen isn’t happening) with that user load assuming you have a decent amount of ram (I have 32gb so you don’t need absurd amounts).
However if you have a lot of users hitting you, 5-6+ or more concurrent streams that all transcode, then you need to start evaluating a discrete gpu (and maybe a significant internet connection bc damn). Alternatively you can suggest your users get something like a ugoos am6b+ flashed with coreelec or a similar setup that can just direct play basically anything but that’s a bit challenging to setup
So then it may be as simple as buying some e waste pc to use a server and using the nas as its intended purpose. Frankly this is probably better, it’s worse power wise but having the storage separate from services has advantages
-
This makes sense
No sense in getting rid of hardware that is working. I’m not familiar with ersatztv but for all the other stuff I am able to handily run it on a 10th gen intel build that is also handling nas duties fwiw. And some stuff is not ideal (cctv is handled via blue iris, which runs in windows VM, everything else is docker)
for the gpu it really depends on your needs. How many users is the big one. If you have at most 2-4 concurrent users and that is an uncommon scenario the gpu is a waste of power, money, and thermal management. Igpu will sip power and transcode (depending on library content, again av1/vp9 on a 10th gen isn’t happening) with that user load assuming you have a decent amount of ram (I have 32gb so you don’t need absurd amounts).
However if you have a lot of users hitting you, 5-6+ or more concurrent streams that all transcode, then you need to start evaluating a discrete gpu (and maybe a significant internet connection bc damn). Alternatively you can suggest your users get something like a ugoos am6b+ flashed with coreelec or a similar setup that can just direct play basically anything but that’s a bit challenging to setup
So then it may be as simple as buying some e waste pc to use a server and using the nas as its intended purpose. Frankly this is probably better, it’s worse power wise but having the storage separate from services has advantages
So I'm expecting a max of 5 concurrent users, but most wouldn't need transcoding. The real hiccup (brace yourself) is a 720p CRT and (assuming I get transcoding to work well) a 480p CRT. I'm pretty novice to PC specs outside of the "buy whatever you can afford for gaming" mindset, so any suggestions there are welcome. My budget is...whatever it takes to not regret the hardware years from now. My last build was $2k for reference
-
That's my goal: use the NAS as a NAS, use a computer for containers and the like. I'm using Seagate Exos for the NAS exclusively
Cool, well then I can at least share what I went with that has worked really well: GMKtech N100 NUC from Aliexpress with 16 GB of RAM.
It's hosting Jellyfin with transcoding, PiHole, Home Assistant, Heimdal, a Valheim server and loads of other small LXC's in Proxmox.
I don't think I've ever seen it break a sweat.
The NAS holds the .arr stack and Qbit, but that's it.I cannot speak to the longevity of it, but I repasted the CPU once I got it and it's chilling below 45 degrees all day long, so I expect it to last for many years. I also enabled C-states to get idle consumption as low as possible, around 7-8W.
Best of luck with whatever setup you end up with mate!
-
It's 4 bays, and we're eating that space up quicker than I imagined
wrote last edited by [email protected]I'm in the same boat. My 2 bay Synology is enough for my storage, but the apps and Docker containers are bringing it to its knees. I'm keeping the Synology and adding on a mini PC to run docker containers.
Found a mini PC with an i5-12500T for £240.
I'm a noob at this stuff, but I'm hyped to get more stability in what I host and start using more demanding services.
