Help Reviewing My Server Setup?
-
Does your storage include any kind of RAID? If not then that's something I'd personally add in to the mix to avoid interruptions for the service. Also 32 gig of RAM is not much, so don't use ZFS on proxmox, it eats up your memory and if you run out everything is stupidly slow (personal experience speaking here, my proxmox server has 32gig as well).
Also, that's quite a lot of stuff to maintain, but you do you. Personally I would not like that big stack to maintain for my everyday needs, but I have wife, kids, kids hobbies and a ton of other stuff going on so I have barely enough personal capacity to run my own proxmox, pihole, immich and HomeAssistant and none of those are in perfect condition. Specially the HA setup badly needs some TLC.
And then there's the obvious. Personal mail server on a home grade uplink is a beast of it's on to manage and if you really don't know what you're getting into I'd recommend against it. And I'm advocating every mail server which is not owned by alphabet/microsoft/apple/etc. It's just a complicated thing to do right and email is quite essential thing for everyday life today, so be aware. If you know what's coming up (or are willing to eat up the mistakes and learn from them) then by all means, go for it. If not, then I'd suggest paying for someone to make it happen.
And then the backups. I've made the mistake few times where I thought it'd be fine to set up backups at some point in the future. And that has bit me in the rear. You either have backups on the pipeline coming Very Soon(tm) or you lose your data. And even if it's coming Very Soon, you'll still risk losing your data.
Plus with backups, if you don't test recovery from them then you don't have backups. Altough for a home gamer it's often a bit much to ask for a blank slate recovery, so at least I've settled on the scenario where I know for sure I can recover from any disaster happening in the home lab without testing as I don't have enough spare hardware to run that test fully.
Beyond that, just have fun. Recently I ran into an issue where my proxmox server needed some hardware maintenance/changes and that took my pihole-server down, so whole LAN was out of DNS services. No tthe end of the world for me, but a problem anyways and I've been planning for a remedy against that, but haven't yet done anyting concrete for it.
-
Just remember the KISS principal: Keep It Simple, Stupid
Keep the NAS as a NAS, and I would honestly trim all of that down to a clustered hypervisor setup (like Proxmox) with dedicated VMs to run each stack. That way if you need to take a machine down for whatever reason, you can migrate its VMs/containers to another machine, with minimal downtime, so you can do whatever it is you need to do with said machine.
Full disclosure: this is what I do. I was in your shoes before.
-
I think a lot comes down to usage. It just depends whether you connect 1 camera to Frigate, or 6. And if you enable some AI features. Whether you download a lot of TV series or a few and delete old stuff. Or use ZFS or other demanding things. I personally like to keep the amount of servers low. So I probably wouldn't buy server 2 and try running those services on 1 as well. I'm not sure. You did a good job seperating the stuff. And I think you got some good advice already. I'd add more harddisks, 6TB wouldn't do it for me. And some space for backups. But you can always keep an eye on actual resource usage and just buy RAM and harddisks as needed. As long as your servers have some slots left for future upgrades.
-
I'm interested in why you chose the i5 for the automation, rather than the video server?
I'm no expert, but things like transcoding (or even just re-encoding) take a lot of grunt, which it seems the i5 would be good for.
The i3 would be good for more constant, lower power tasks like automation.
At least, that's my thoughts, happy to be shown your reasoning..
-
Thanks so much for the response! Server 1 has been running strong for quite a while with six cameras on Frigate and very little CPU usage. I do have a ZFS pool on Server 1– this is the first I'm hearing that it requires more resource overhead... Could you elaborate a bit?
6TB is just to start, and I fully intend to upgrade both RAM and storage as I need it.
-
Thanks for the advice! I have local backups, but in looking to use more self-hosted services, I want an offsite backup in case, say, my house burns down. That way I don't lose all my photos, etc.
I've decided to forego a mail server– you're not the first to tell me it's very difficult to maintain and setup properly, ha!
-
I'm not an expert, how many Raspberry Pi 4s does this translate to?
-
I would put truenas on the NAS, also put a VM on truenas with 16-24G of RAM.
Create a kubernetes or docker swarm cluster with server 1 and the nas vm and just have everything as containers. This way you just have one resource pool, and the containers will be started wherever there are enough resources available. The containers will mount NFS shares from truenas which truenas will create automatically as ZFS datasets. ZFS supports snapshots.
-
ZFS is an enterprise software RAID, and 1:1 RAM to TB is the minimum recommended requirement for a production server (e.g. enterprise implementations).
I've seen many users stating they have far far less than 1:1 without issues. I recall a r/DataHoarder user saying they have 100+ TB's and only 16 or 32GB RAM, which is not fully utilized, so it all depends on your usage profile and the size/scale of r/w ops occurring during peak periods.
-
I personally would avoid LXC. That seems to be a hot take but in my experience it is better to run docker/podman in a few VMs.
-
Cost wise? 1 or 2 /s
-
The GPU should be the same for most Intel systems
-
I wouldn't do that unless you have lots of money to blow on crazy hardware. Running separate virtual machines is very inefficient. Instead, run a few virtual machines with a few services in each. I would separate it out into classes based on the load and use case.
-
Not everything plays nice in Docker, and there are plenty of those services that also don't need a full VM to operate. LXC is great for those edge cases. Otherwise I agree, a few VMs for various Docker stacks is the way to go.
-
...really? I run most of my services in an LXC, and have for a while without issue.
-
I had the i5 prior to getting a NAS, and use it for Frigate. The i3 is just what came with the NAS box.
-
Maybe I'm doing it wrong then. I run LXC but has always been a much worse experience. Boot times are terrible and the controls that work for VMs don't work as well for LXC. You also can live transfer which is problematic for me.
-
Instead, run a few virtual machines with a few services in each.
That's what I meant, I guess it wasn't very clear. When I say "stack", I mean multiple services.
-
I'm also considering UnRaid instead of Proxmox for a NAS OS.
NAS just has no meaning anymore?
-
Personally I would keep it simple and just run a separate NAS and run all your services in containers across the devices best suited to them. The i3 is not going to manage for Jellyfin while sharing those other services. I tried running it on an N100 and had to move it to a beefier machine.
If you mount a NAS storage for hosting the container data, you can move them between machines with minimal issues. Just make sure you run services using a docker-compose for them and keep them on the NAS.
You completely negate the need for VMs and their overhead, can still snapshot the machine if you run debian as the OS there is timeshift. Other distros have similar.
