Homelab upgrade - "Modern" alternatives to NFS, SSHFS?
-
I have my NFS storage mounted via 2.5G and use qcow2 disks. It is slow to snapshot...
Maybe I understand your question wrong?
-
Generally yes, but it can be useful as a learning thing. A lot of my homelab use is for purposes of practicing with different techs in a setting where if it melts down it's just your stuff. At work they tend to take offense of you break prod.
-
I tried it once. NFSv4 isn't simple like NFSv3 is. Fewer systems support it too.
-
Are you having trouble reading context?
No, I'm not applying 2005 security, I'm saying NFS hasn't evolved much since 2005, so throw it in a dedicated link by itself with no other traffic and call it a day.
Yes, iscsi allows the use of mounted luns as datastores like any other, you just need to use the user space iscsi driver and tools so that iscsi-ls is available. Do not use the kernel driver and args. This is documented in many places.
If you're gonna make claims to strangers on the internet, make sure you know what you're talking about first.
-
I agree, but it’s clear that OP doesn’t want a real solution, because those apparently are boring. Instead, they want to try something new. NVMe/TCP is something new. And it still allows for having VMs on one system and storage on another, so it’s not entirely off topic.
-
NFS doesn't do snapshotting, which is what I assumed that you meant and I'd guess ShortN0te also assumed.
If you're talking about qcow2 snapshots, that happens at the qcow2 level. NFS doesn't have any idea that qemu is doing a snapshot operation.
On a related note: if you are invoking a VM using a filesystem images stored on an NFS mount, I would be careful, unless you are absolutely certain that this is safe for the version of NFS and the specific caching options for both NFS and qemu that you are using.
I've tried to take a quick look. There's a large stack involved, and I'm only looking at it quickly.
To avoid data loss via power loss, filesystems -- and thus the filesystem images backing VMs using filesystems -- require write ordering to be maintained. That is, they need to have the ability to do a write and have it go to actual, nonvolatile storage prior to any subsequent writes.
At a hard disk protocol level, like for SCSI, there are BARRIER operations. These don't force something to disk immediately, but they do guarantee that all writes prior to the BARRIER are on nonvolatile storage prior to writes subsequent to it.
I don't believe that Linux has any userspace way for an process to request a write barrier. There is not an
fwritebarrier()call. This means that the only way to impose write ordering is to call fsync()/sync() or use similar-such operations. These force data to nonvolatile storage, and do not return until it is there. The downside is that this is slow. Programs that are frequently doing such synchronizations cannot issue writes very quickly, and are very sensitive to latency to their nonvolatile storage.From the
qemu(1)man page:By default, the cache.writeback=on mode is used. It will report data writes as completed as soon as the data is present in the host page cache. This is safe as long as your guest OS makes sure to correctly flush disk caches where needed. If your guest OS does not handle volatile disk write caches correctly and your host crashes or loses power, then the guest may experience data corruption. For such guests, you should consider using cache.writeback=off. This means that the host page cache will be used to read and write data, but write notification will be sent to the guest only after QEMU has made sure to flush each write to the disk. Be aware that this has a major impact on performance.I'm fairly sure that this is a rather larger red flag than it might appear, if one simply assumes that Linux must be doing things "correctly".
Linux doesn't guarantee that a write to position A goes to disk prior to a write to position B. That means that if your machine crashes or loses power, with the default settings, even for drive images sorted on a filesystem on a local host, with default you can potentially corrupt a filesystem image.
https://docs.kernel.org/block/blk-mq.html
Note
Neither the block layer nor the device protocols guarantee the order of completion of requests. This must be handled by higher layers, like the filesystem.
POSIX does not guarantee that write() operations to different locations in a file are ordered.
https://stackoverflow.com/questions/7463925/guarantees-of-order-of-the-operations-on-file
So by default -- which is what you might be doing, wittingly or unwittingly -- if you're using a disk image on a filesystem,
qemusimply doesn't care about write ordering to nonvolatile storage. It does writes. it does not care about the order in which they hit the disk. It is not callingfsync()or using analogous functionality (likeO_DIRECT).NFS entering the picture complicates this further.
https://www.man7.org/linux/man-pages/man5/nfs.5.html
The sync mount option
The NFS client treats the sync mount option differently than some
other file systems (refer to mount(8) for a description of the
generic sync and async mount options). If neither sync nor async
is specified (or if the async option is specified), the NFS client
delays sending application writes to the server until any of these
events occur:Memory pressure forces reclamation of system memory resources. An application flushes file data explicitly with sync(2), msync(2), or fsync(3). An application closes a file with close(2). The file is locked/unlocked via fcntl(2). In other words, under normal circumstances, data written by an application may not immediately appear on the server that hosts the file. If the sync option is specified on a mount point, any system call that writes data to files on that mount point causes that data to be flushed to the server before the system call returns control to user space. This provides greater data cache coherence among clients, but at a significant performance cost. Applications can use the O_SYNC open flag to force application writes to individual files to go to the server immediately without the use of the sync mount option.So, strictly-speaking, this doesn't make any guarantees about what NFS does. It says that it's fine for the NFS client to send nothing to the server at all on write(). The only time a write() to a file makes it to the server, if you're using the default NFS mount options. If it's not going to the server, it definitely cannot be flushed to nonvolatile storage.
Now, I don't know this for a fact -- would have to go digging around in the NFS client you're using. But it would be compatible with the guarantees listed, and I'd guess that probably, the NFS client isn't keeping a log of all the write()s and then replaying them in order. If it did so, for it to meaningfully affect what's on nonvolatile storage, the NFS server would have to fsync() the file after each write being flushed to nonvolatile storage. Instead, it's probably just keeping a list of dirty data in the file, and then flushing it to the NFS server at close().
That is, say you have a program that opens a file filled with all '0' characters, and does:
- write '1' to position 1.
- write '1' to position 5000.
- write '2' to position 1.
- write '2' to position 5000.
At close() time, the NFS client probably doesn't flush "1" to position 1, then "1" to position 5000, then "2" to position 1, then "2" to position 5000. It's probably just flushing "2" to position 1, and then "2" to position 5000, because when you close the file, that's what's in the list of dirty data in the file.
The thing is that unless the NFS client retains a log of all those write operations, there's no way to send the writes to the server in a way that avoid putting the file into a corrupt state if power is lost. It doesn't matter whether it writes the "2" at position 1 or the "2" at position 5000. In either case, it's creating a situation where, for a moment, one of those two positions has a "0", and the other has a "2". If there's a failure at that point -- the server loses power, the network connection is severed -- that's the state in which the file winds up in. That's a state that is inconsistent, should never have arisen. And if the file is a filesystem image, then the filesystem might be corrupt.
So I'd guess that at both of those two points in the stack -- the NFS client writing data to the server, and the server block device scheduler, permit inconsistent state if there's no fsync()/sync()/etc being issued, which appears to be the default behavior for
qemu. And running on NFS probably creates a larger window for a failure to induce corruption.It's possible that using qemu's iSCSI backend avoids this issue, assuming that the iSCSI target avoids reordering. That'd avoid qemu going through the NFS layer.
I'm not going to dig further into this at the moment. I might be incorrect. But I felt that I should at least mention it, since filesystem images on NFS sounded a bit worrying.
-
You misunderstand. The hypervisor is the client. Stuff higher in the stack only sees raw storage. (By hypervisors I also mean docker and kubernetes) From a security perspective you just set an IP allow list
-
Yeah, I've ended up setting up VLANS in order to not deal with encryption
-
Sure, if you have exactly one client that can access the server and you can ensure physical security of the actual network, I suppose it is fine. Still, those are some severe limitations and show how limited the ancient NFS protocol is, even in version 4.
-
Thanks for digging this "shallow" (lol. What you dug up is equal to my senior technician explaining the full tech stack of a client).
Anyway, I host the system disk on local disk and NFS storage acts as the mass storage for my VMs like my media server for jellyfin).
And I also do daily backups with Veeam Backup and Replication of both my most important files of my media server and the important VMs.
So in case of a data failure it should be more or less fine. -
If i understand you correctly, your Server is accessing the VM disk images via a NFS share?
That does not sound efficient at all.
-
No other easy option I figured out.
Didnt manage to understand iSCSI in the time I was patient with it and was desperate to finish the project and use my stuff.
Thus NFS. -
Wouldnt the sync option also confirm that every write also arrived on the disk?
If you're mounting with the NFS sync option, that'll avoid the "wait until close and probably reorder writes at the NFS layer" issue I mentioned, so that'd address one of the two issues, and the one that's specific to NFS.
-
NFS gives me the best performance. I've tried GlusterFS (not at home, for work), and it was kind of a pain to set up and maintain.
-
As a recently former hpc/supercomputer dork nfs scales really well. All this talk of encryption etc is weird you normally just do that at the link layer if you’re worried about security between systems. That and v4 to reduce some metadata chattiness and gtg. I’ve tried scaling ceph and s3 for latency on 100/200g links. By far NFS is easier than all the rest to scale. For a homelab? NFS and call it a day, all the clustering file systems will make you do a lot more work than just throwing hard into your nfs mount options and letting clients block io while you reboot. Which for home is probably easiest.
-
I agree as well. No reason to not use it. If there were better ways to build an alternative, one would exist.
-
I use Ceph/CephFS myself for my own 671TiB array (382TiB raw used, 252TiB-ish data stored) -- I find it a much more robust and better architected solution than Gluster. It supports distributed block devices (RBD), filesystems (CephFS), and object storage (RGW). NFS is pretty solid though for basic remote mounting filesystems.
