Docker to Kubernetes, ZFS to ... ?
-
Hi all,
I've been running a bunch of services in docker containers using Docker Compose for a while now, with data storage on ZRAID mirrored NVME and/or ZRAID2 HDDs.
I've been thinking about moving from my single server setup to three micro-servers (Intel N150s), both for redundancy, learning, and fun.
Choosing Kubernetes was easy, but I'd like to get some outside opinions on storage. Some examples of how I'm using storage:
- Media and large data storage: Currently on the ZRAID2 HDDs, will stay here but be migrated to a dedicated NAS
- High IO workloads like Postgresql and email: Currently running on the NVMEs
- General low-volume storage: Also currently on NVMEs, but different use case. These are lower IO, like data storage for Nextcloud, Immich, etc
I'm a huge fan of being able to snapshot with ZFS, as I mirror all my data off-site with hourly pushes for some container data, and daily for the rest. I'd like to be able to continue this kind of block-level backups if possible.
Assume I'm a noob at Kubernetes storage (have been reading, but still fresh to me). I'd love to know how others would set up their storage interfaces for this.
I'm trying to understand if there's a way to have the storage "RAIDed" across the drives in the three micro-servers, or if things work differently than I expect. Thanks!
-
System shared this topic on
-
Hi all,
I've been running a bunch of services in docker containers using Docker Compose for a while now, with data storage on ZRAID mirrored NVME and/or ZRAID2 HDDs.
I've been thinking about moving from my single server setup to three micro-servers (Intel N150s), both for redundancy, learning, and fun.
Choosing Kubernetes was easy, but I'd like to get some outside opinions on storage. Some examples of how I'm using storage:
- Media and large data storage: Currently on the ZRAID2 HDDs, will stay here but be migrated to a dedicated NAS
- High IO workloads like Postgresql and email: Currently running on the NVMEs
- General low-volume storage: Also currently on NVMEs, but different use case. These are lower IO, like data storage for Nextcloud, Immich, etc
I'm a huge fan of being able to snapshot with ZFS, as I mirror all my data off-site with hourly pushes for some container data, and daily for the rest. I'd like to be able to continue this kind of block-level backups if possible.
Assume I'm a noob at Kubernetes storage (have been reading, but still fresh to me). I'd love to know how others would set up their storage interfaces for this.
I'm trying to understand if there's a way to have the storage "RAIDed" across the drives in the three micro-servers, or if things work differently than I expect. Thanks!
Glusterfs?
It'll let you turn the storage on each server into a single volume and you can set raid-like mirroring/striping.
-
Hi all,
I've been running a bunch of services in docker containers using Docker Compose for a while now, with data storage on ZRAID mirrored NVME and/or ZRAID2 HDDs.
I've been thinking about moving from my single server setup to three micro-servers (Intel N150s), both for redundancy, learning, and fun.
Choosing Kubernetes was easy, but I'd like to get some outside opinions on storage. Some examples of how I'm using storage:
- Media and large data storage: Currently on the ZRAID2 HDDs, will stay here but be migrated to a dedicated NAS
- High IO workloads like Postgresql and email: Currently running on the NVMEs
- General low-volume storage: Also currently on NVMEs, but different use case. These are lower IO, like data storage for Nextcloud, Immich, etc
I'm a huge fan of being able to snapshot with ZFS, as I mirror all my data off-site with hourly pushes for some container data, and daily for the rest. I'd like to be able to continue this kind of block-level backups if possible.
Assume I'm a noob at Kubernetes storage (have been reading, but still fresh to me). I'd love to know how others would set up their storage interfaces for this.
I'm trying to understand if there's a way to have the storage "RAIDed" across the drives in the three micro-servers, or if things work differently than I expect. Thanks!
That would be block storage like glusterfs or ceph, or object storage like minio.
You could also use ZFS to provide PVCs for your Pods, with openebs.
If the mini-servers don’t have hardware redundancy, I’d stick to Replicated Volumes only…
-
Hi all,
I've been running a bunch of services in docker containers using Docker Compose for a while now, with data storage on ZRAID mirrored NVME and/or ZRAID2 HDDs.
I've been thinking about moving from my single server setup to three micro-servers (Intel N150s), both for redundancy, learning, and fun.
Choosing Kubernetes was easy, but I'd like to get some outside opinions on storage. Some examples of how I'm using storage:
- Media and large data storage: Currently on the ZRAID2 HDDs, will stay here but be migrated to a dedicated NAS
- High IO workloads like Postgresql and email: Currently running on the NVMEs
- General low-volume storage: Also currently on NVMEs, but different use case. These are lower IO, like data storage for Nextcloud, Immich, etc
I'm a huge fan of being able to snapshot with ZFS, as I mirror all my data off-site with hourly pushes for some container data, and daily for the rest. I'd like to be able to continue this kind of block-level backups if possible.
Assume I'm a noob at Kubernetes storage (have been reading, but still fresh to me). I'd love to know how others would set up their storage interfaces for this.
I'm trying to understand if there's a way to have the storage "RAIDed" across the drives in the three micro-servers, or if things work differently than I expect. Thanks!
-
Hi all,
I've been running a bunch of services in docker containers using Docker Compose for a while now, with data storage on ZRAID mirrored NVME and/or ZRAID2 HDDs.
I've been thinking about moving from my single server setup to three micro-servers (Intel N150s), both for redundancy, learning, and fun.
Choosing Kubernetes was easy, but I'd like to get some outside opinions on storage. Some examples of how I'm using storage:
- Media and large data storage: Currently on the ZRAID2 HDDs, will stay here but be migrated to a dedicated NAS
- High IO workloads like Postgresql and email: Currently running on the NVMEs
- General low-volume storage: Also currently on NVMEs, but different use case. These are lower IO, like data storage for Nextcloud, Immich, etc
I'm a huge fan of being able to snapshot with ZFS, as I mirror all my data off-site with hourly pushes for some container data, and daily for the rest. I'd like to be able to continue this kind of block-level backups if possible.
Assume I'm a noob at Kubernetes storage (have been reading, but still fresh to me). I'd love to know how others would set up their storage interfaces for this.
I'm trying to understand if there's a way to have the storage "RAIDed" across the drives in the three micro-servers, or if things work differently than I expect. Thanks!
For starting out I would suggest Longhorn, I've had several issues with it but maybe they've been fixed. I also used Rook Ceph which is overkill but neat to play with.
-
System shared this topic on
