From Docker with Ansible to k3s: I don't get it...
-
curl -sfL https://get.k3s.io/ | sh -
Never, ever install anything this way. The trend of "just run this shell script off the internet" is a menace. You don't know what that script does, what repositories it may add, what it may install, whether somebody is typo-squatting the URL and you're running something else, etc.
It's just a bad idea. If you disagree then I have one question - how would you uninstall k3s after you ran that blackbox?
Yes, just running a random script from the internet is a very bad idea. You should also not copy and paste the command from above, since I'm only a random lemmy user.
Nevertheless, if you trust k3s, and they promote this command on the official website (make sure it's the official one) you can use it.
As you want to install k3s, I'm going to assume you trust k3s.If you want to review the script, go for it. And you should, I agree.
I for myself reviewed (or at least looked over it) when I used the script for myself.For the uninstallment: just follow the instructions on the official website and run
/usr/local/bin/k3s-uninstall.shsource -
Yes, just running a random script from the internet is a very bad idea. You should also not copy and paste the command from above, since I'm only a random lemmy user.
Nevertheless, if you trust k3s, and they promote this command on the official website (make sure it's the official one) you can use it.
As you want to install k3s, I'm going to assume you trust k3s.If you want to review the script, go for it. And you should, I agree.
I for myself reviewed (or at least looked over it) when I used the script for myself.For the uninstallment: just follow the instructions on the official website and run
/usr/local/bin/k3s-uninstall.shsourcewrote last edited by [email protected]I really want to push back on the entire idea that it's okay to distribute software via a
curl | shcommand. It's a bad practice. I shouldn't be reading 100's of lines of shell script to see what sort of malarkey your installer is going to do to my system. This application creates an uninstall script. Neat. Many don't.Of the myriad ways to distribute Linux software (deb, rpm, snap, flatpak, AppImage) an unstructured shell script is by far the worst.
-
curl -sfL https://get.k3s.io/ | sh -
Never, ever install anything this way. The trend of "just run this shell script off the internet" is a menace. You don't know what that script does, what repositories it may add, what it may install, whether somebody is typo-squatting the URL and you're running something else, etc.
It's just a bad idea. If you disagree then I have one question - how would you uninstall k3s after you ran that blackbox?
https://docs.k3s.io/installation/uninstall
There is also a k3s option for Nixos, which removes the security and side-affect risks of running a random bash script installer.
-
Hey!
I have been using Ansible to deploy Dockers for a few services on my Raspberry Pi for a while now and it's working great, but I want to learn MOAR and I need help...Recently, I've been considering migrating to bare metal K3S for a few reasons:
- To learn and actually practice K8S.
- To have redundancy and to try HA.
- My RPi are all already running on MicroOS, so it kind of make sense to me to try other SUSE stuff (?)
- Maybe eventually being able to manage my two separated servers locations with a neat k3s + Tailscale setup!
Here is my problem: I don't understand how things are supposed to be done. All the examples I find feel wrong.
More specifically:- Am I really supposed to have a collection of small yaml files for everything, that I use with
kubectl apply -f?? It feels wrong and way too "by hand"! Is there a more scripted way to do it? Should I stay with everything in Ansible ?? - I see little to no example on how to deploy the service containers I want (pihole, navidrome, etc.) to a cluster, unlike docker-compose examples that can be found everywhere. Am I looking for the wrong thing?
- Even official doc seems broken. Am I really supposed to run many helm commands (some of them how just fails) and try and get ssl certs just to have Rancher and its dashboard ?!
I feel that having a K3S + Traefik + Longhorn + Rancher on MicroOS should be straightforward, but it's really not.
It's very much a noob question, but I really want to understand what I am doing wrong. I'm really looking for advice and especially configuration examples that I could try to copy, use and modify!
Thanks in advance,
Cheers!
And this is why I do not like K8s at all. The only reason to use it is to have something on your CV. Besides that, Docker Swarm and Hashicorp Nomad feel a lot better and are a lot easier to manage.
-
I really want to push back on the entire idea that it's okay to distribute software via a
curl | shcommand. It's a bad practice. I shouldn't be reading 100's of lines of shell script to see what sort of malarkey your installer is going to do to my system. This application creates an uninstall script. Neat. Many don't.Of the myriad ways to distribute Linux software (deb, rpm, snap, flatpak, AppImage) an unstructured shell script is by far the worst.
I think that distributing general software via
curl | shis pretty bad for all the reasons that curl sh is bad and frustrating.But I do make an exception for "platforms" and package managers. The question I ask myself is: "Does this software enable me to install more software from a variety of programming languages?"
If the answer to that question is yes, which is is for k3s, then I think it's an acceptable exception.
curl | shis okay for bootstrapping things like Nix on non Nix systems, because then you get a package manager to install various versions of tools that would normally try to get you to install themselves withcurl | bashbut then you can use Nix instead.K3s is pretty similar, because Kubernetes is a whole platform, with it's own package manager (helm), and applications you can install. It's especially difficult to get the latest versions of Kubernetes on stable release distros, as they don't package it at all, so getting it from the developers is kinda the only way to get it installed.
Relevant discussion on another thread: https://programming.dev/post/33626778/18025432
One of my frustrations that I express in the linked discussion is that it's "developers" who are making bash scripts to install. But k3s is not just developers, it's made by Suse who has their own distro, OpenSuse, using OpenSuse tooling. It's "packagers" making k3s and it's install script, and that's another reason why I find it more acceptable.
-
Hey!
I have been using Ansible to deploy Dockers for a few services on my Raspberry Pi for a while now and it's working great, but I want to learn MOAR and I need help...Recently, I've been considering migrating to bare metal K3S for a few reasons:
- To learn and actually practice K8S.
- To have redundancy and to try HA.
- My RPi are all already running on MicroOS, so it kind of make sense to me to try other SUSE stuff (?)
- Maybe eventually being able to manage my two separated servers locations with a neat k3s + Tailscale setup!
Here is my problem: I don't understand how things are supposed to be done. All the examples I find feel wrong.
More specifically:- Am I really supposed to have a collection of small yaml files for everything, that I use with
kubectl apply -f?? It feels wrong and way too "by hand"! Is there a more scripted way to do it? Should I stay with everything in Ansible ?? - I see little to no example on how to deploy the service containers I want (pihole, navidrome, etc.) to a cluster, unlike docker-compose examples that can be found everywhere. Am I looking for the wrong thing?
- Even official doc seems broken. Am I really supposed to run many helm commands (some of them how just fails) and try and get ssl certs just to have Rancher and its dashboard ?!
I feel that having a K3S + Traefik + Longhorn + Rancher on MicroOS should be straightforward, but it's really not.
It's very much a noob question, but I really want to understand what I am doing wrong. I'm really looking for advice and especially configuration examples that I could try to copy, use and modify!
Thanks in advance,
Cheers!
I've thought about k8s, but there is so much about Docker that I still don't fully know.
-
I think that distributing general software via
curl | shis pretty bad for all the reasons that curl sh is bad and frustrating.But I do make an exception for "platforms" and package managers. The question I ask myself is: "Does this software enable me to install more software from a variety of programming languages?"
If the answer to that question is yes, which is is for k3s, then I think it's an acceptable exception.
curl | shis okay for bootstrapping things like Nix on non Nix systems, because then you get a package manager to install various versions of tools that would normally try to get you to install themselves withcurl | bashbut then you can use Nix instead.K3s is pretty similar, because Kubernetes is a whole platform, with it's own package manager (helm), and applications you can install. It's especially difficult to get the latest versions of Kubernetes on stable release distros, as they don't package it at all, so getting it from the developers is kinda the only way to get it installed.
Relevant discussion on another thread: https://programming.dev/post/33626778/18025432
One of my frustrations that I express in the linked discussion is that it's "developers" who are making bash scripts to install. But k3s is not just developers, it's made by Suse who has their own distro, OpenSuse, using OpenSuse tooling. It's "packagers" making k3s and it's install script, and that's another reason why I find it more acceptable.
Microk8s manages to install with a snap. I know that snap is "of the devil" around these parts but it's still better than a custom bash script.
Custom bash scripts will always be worse than any alternative.
-
Microk8s manages to install with a snap. I know that snap is "of the devil" around these parts but it's still better than a custom bash script.
Custom bash scripts will always be worse than any alternative.
I've tried snap, juju, and Canonical's suite. They were uniquely frustrating and I'm not interested in interacting with them again.
The future of installing system components like k3s on generic distros is probably systemd sysexts, which are extension images that can be overlayed onto a base system. It's designed for immutable distros, but it can be used on any standard enough distro.
There is a k3s sysext, but it's still in the "bakery". Plus sysext isn't in stable release distros anyways.
Until it's out and stable, I'll stick to the one time bash script to install Suse k3s.
-
I've tried snap, juju, and Canonical's suite. They were uniquely frustrating and I'm not interested in interacting with them again.
The future of installing system components like k3s on generic distros is probably systemd sysexts, which are extension images that can be overlayed onto a base system. It's designed for immutable distros, but it can be used on any standard enough distro.
There is a k3s sysext, but it's still in the "bakery". Plus sysext isn't in stable release distros anyways.
Until it's out and stable, I'll stick to the one time bash script to install Suse k3s.
You're welcome to make whatever bad decisions you like. I can manage snaps with standard tooling. I can install, update, remove them with simple ansible scripts in a standard way.
Bash installers are bad. End of.
-
You're welcome to make whatever bad decisions you like. I can manage snaps with standard tooling. I can install, update, remove them with simple ansible scripts in a standard way.
Bash installers are bad. End of.
Canonical's snap use a proprietary backend, and comes at a risk of vendor lock in to their ecosystem.
The bash installer is fully open source.
You can make the bad decision of locking yourself into a closed ecosystem, but many sensible people recognize that snap is "of the devil" for a good reason.
-
Or Kustomize, though I prefer Helm.
Then helmfile might be worth checking out
-
I'll post more later (reply here to remind me), but I have your exact setup. It's a great way to learn k8s and yes, it's going to be an uphill battle for learning - but the payoff is worth it. Both for your professional career and your homelab. It's the big leagues.
For your questions, no to all of them. Once you learn some of it the rest kinda falls together.
I'm going into a meeting, but I'll post here with how I do it later. In the mean time, pick one and only one container you want to get started with. Stateless is easier to start with compared to something that needs volumes. Piece by piece brick by brick you will add more to your knowledge and understanding. Don't try to take it all on day one. First just get a container running. Then access via a port and http. Then proxy. Then certs. Piece by piece, brick by brick. Take small victories, if you try to say "tomorrow everything will be on k8s" you're setting yourself up for anger and frustration.
@[email protected]
Edit: To help out I would do these things in these steps, note that steps are not equal in length, and they are not complete - but rather to help you get started without burning out on your journey. I recommend just taking each one, and when you get it working rather than jumping to the next one, instead taking a break, having a drink, and celebrating that you got it up and running.-
Start documenting everything you do. The great thing about kubernetes is that you can restart from scratch if you have written everything down. I would start a new git repository with a README that contains every command you ran, what it did, and why you did it. Assume that you will be tearing down your cluster and rebuilding it - in fact I would even recommend that. Treat this first cluster as your testing grounds, and then you won't feel crappy spinning up temporary resources. Then, you can rebuild it and know that you did a great job - and you'll feel confident in rebuilding in case of hardware failure.
-
Get the sample nginx pod up and running with a service and deployment. Simply so you can
curlthe IP of your main node and port, and see the response. This I assume you have played with already. -
Point DNS to your main node, get the nginx pod with
http://your.dns.tld:PORT. This should be the same as anything you've done with docker before. -
Convert the yaml to a helm chart as other have said, but don't worry about "templating" yet, get comfortable with
helm install,helm upgrade -i, andhelm uninstall. Understand what each one does and how they operate. Then go back and template, upgrade-ing after each change to understand how it works. It's pretty standard to template the image and tag for example so it's easy to upgrade them. There's a million examples online, but don't go overboard, just do the basics. My (template values.yaml) usually looks like:
<<servicename>> name: <<servicename>> image: repository: path/to/image tag: v1.1.1 network: port: 8888Just keep it simple for now.
- Decide on your proxy service. Traefik as you see comes out of the box. I personally use
istio. I can go into more details why later, but I like that I can create a "VirtualService" for "$appname.my.custom.tld` and it will point to it. - Implement your proxy service, and get the (http only still) app set up. Set up something like
nginx.your.tldand be able to curlhttp://nginx.your.tldand see that it routes properly to your sample nginx service. Congrats, this is a huge one. - Add the CertManager chart. This will set it up so you can create
Certificatetypes in k8s. You'll need to use the proxy in the previous step to route the /.well-known endpoints on the http port from the open web to cert-manager, for Istio this was another virtual service on the gateway - I assume Traefic would have something similar to "route all traffic on port 80 that starts with /.well-known to this service". Then, in your nginx helm chart, add in a Certificate type for your nginx endpoint,nginx.your.tld, and wait for it to be successfully granted. With Istio, this is all I need now to finally curlhttps://nginx.your.tld!
At this point you have routing, ports, and https set up. Have 2 drinks after this one. You can officially deploy any stateless service at this point.
Now, the big one, stateful. Longhorn is a bear, there are a thousand caveats to it.
Step one is where are your backups going to go. This can be a simple NFS/SMB share on a local server, it can be an s3 endpoint, but seriously this is step 1. Backups are critical with longhorn. You will fuck up Longhorn - multiple times. Losing these backups means losing all configs to all of your pods, so step one is to decide on your stable backup location.
Now, read the Longhorn install guide: https://longhorn.io/docs/1.9.0/deploy/install/. Do not skip reading the install guide. There are incredibly important things in there that I regretted glossing over that would have saved me. (Like setting up backups first).
The way I use longhorn is to create a PV in longhorn, and then the PVC (you can look up what both of these are later). Then I use Helm to set what the PVC name is to attach it to my pod. Try and do this with another sample pod. You are still not ready to move production things over yet, so just attach it to nginx.
execinto it, write some data into the pvc. Helm uninstall. See what happens in longhorn. Helm install. Does your PVC reattach? Exec in, is your data still there? Learn how it works. I fully expect you to ping me with questions at this point, don't worry, I'll be here.Longhorn will take time in learning, give yourself grace. Also after you feel comfortable with it, you'll need to start moving data from your old docker setup to Longhorn, and that too will be a process. You'll get there though. Just start with some of your lower priority projects, and migrate them one by one.
After all of this, there is still more. You can automount smb/nfs shares directly into pods for media or anything. You can pass in GPUs - or I even pass in some USB devices. You can encrypt your longhorn things, you can manage secrets with your favorite secret manager. There's thousands of things you'll be able to do. I wish you luck, and feel free to ping me here or on Matrix (@[email protected]) if you ever need an ear. Good luck!
Great writeup.
-
-
Hey!
I have been using Ansible to deploy Dockers for a few services on my Raspberry Pi for a while now and it's working great, but I want to learn MOAR and I need help...Recently, I've been considering migrating to bare metal K3S for a few reasons:
- To learn and actually practice K8S.
- To have redundancy and to try HA.
- My RPi are all already running on MicroOS, so it kind of make sense to me to try other SUSE stuff (?)
- Maybe eventually being able to manage my two separated servers locations with a neat k3s + Tailscale setup!
Here is my problem: I don't understand how things are supposed to be done. All the examples I find feel wrong.
More specifically:- Am I really supposed to have a collection of small yaml files for everything, that I use with
kubectl apply -f?? It feels wrong and way too "by hand"! Is there a more scripted way to do it? Should I stay with everything in Ansible ?? - I see little to no example on how to deploy the service containers I want (pihole, navidrome, etc.) to a cluster, unlike docker-compose examples that can be found everywhere. Am I looking for the wrong thing?
- Even official doc seems broken. Am I really supposed to run many helm commands (some of them how just fails) and try and get ssl certs just to have Rancher and its dashboard ?!
I feel that having a K3S + Traefik + Longhorn + Rancher on MicroOS should be straightforward, but it's really not.
It's very much a noob question, but I really want to understand what I am doing wrong. I'm really looking for advice and especially configuration examples that I could try to copy, use and modify!
Thanks in advance,
Cheers!
You're right to be reluctant to apply everything by hand. K3s has a built-in feature that watches a directory and applies the manifests automatically:
https://docs.k3s.io/installation/packaged-componentsThis can be used to install Helm charts in a declarative way as well:
https://docs.k3s.io/helmIf you want to keep your solution agnostic to the kubernetes environment, I would recommend that you try ArgoCD (or FluxCD, but I never tried it so YMMV).
-
Hey!
I have been using Ansible to deploy Dockers for a few services on my Raspberry Pi for a while now and it's working great, but I want to learn MOAR and I need help...Recently, I've been considering migrating to bare metal K3S for a few reasons:
- To learn and actually practice K8S.
- To have redundancy and to try HA.
- My RPi are all already running on MicroOS, so it kind of make sense to me to try other SUSE stuff (?)
- Maybe eventually being able to manage my two separated servers locations with a neat k3s + Tailscale setup!
Here is my problem: I don't understand how things are supposed to be done. All the examples I find feel wrong.
More specifically:- Am I really supposed to have a collection of small yaml files for everything, that I use with
kubectl apply -f?? It feels wrong and way too "by hand"! Is there a more scripted way to do it? Should I stay with everything in Ansible ?? - I see little to no example on how to deploy the service containers I want (pihole, navidrome, etc.) to a cluster, unlike docker-compose examples that can be found everywhere. Am I looking for the wrong thing?
- Even official doc seems broken. Am I really supposed to run many helm commands (some of them how just fails) and try and get ssl certs just to have Rancher and its dashboard ?!
I feel that having a K3S + Traefik + Longhorn + Rancher on MicroOS should be straightforward, but it's really not.
It's very much a noob question, but I really want to understand what I am doing wrong. I'm really looking for advice and especially configuration examples that I could try to copy, use and modify!
Thanks in advance,
Cheers!
I use Kube everyday for work but I would recomend you to not use it. It's complicated to answer problems you don't care about. How about docker swarm, or podman services ?
