Testing vs Prod

[n0x0n@lemmy.ml]

Production is my testing lab, but only in my homelab ! I guess I don't care to perfectly secure my services (really dumb and easy passwords, no 2fa, not hiding plain sight passwords....) because I'm not directly exposing them to the web and accessing them externally via Wireguard ! That's really bad practice though, but any time soon will probably clean up that mess, but right now I can't, I have to cook some eggs...

There are 2 things though I actually do have some more complex workflow:

• Rather complex incremental automated backup script for my docker container volumes, databases, config files, compose files.

• Self-hosted mini-CA to access all my services via a nice .lab domain and get rid of that pesky warning on my devices.

I always do some tests if my backups are working on a VM on my personal desktop computer, because no backup means that all those years of tinkering for nothing... This will bring up some nasty depression...

[jovialsodium@lemmy.sdf.org]

Nope. I fiddle until it does what I want. If the thing I'm working on is complex or I'm struggling with it I'll keep versions of configs. And I back up working configs via an rsync. Which, isn't a particularly robust solution but I'm content with it for my needs.

[avidamoeba@lemmy.ca]

Sir, every professional developer knows there's never time and people to maintain the testing environment so testing is done in production! That testing environment you're dreaming of is missed shareholder value.

[marauding_gibberish142@lemmy.dbzer0.com]

I use testing, prod and stale. Stale is simply one version behind prod in case I see something in prod I need to roll back

[lorentz@feddit.it]

I don't have a testing environment, but essentially all my services are on docker saving their data in a directory mounted on the local filesystem. The dockerfile reads the sha version of the image from an env file. I have a shell script which:

1. Triggers a new btrfs snapshot of the volume containing everyithing
2. Pulls the new docker images and stores their hashes in the env file
3. Restarts all the containers.

if a new Docker version is broken rolling back is as simple as copying the old version in the env file and recreating the container. If data gets corrupted I can just copy the last working status from an old snaphot.

The whole os is on a btrfs volume which is snapshotted regularly, so ideally if an update fucks it up beyond recovery I can always boot from a rescue image and restore an old snapshot. But I honestly feel this is extra precaution: in years that I run debian on all my computers, it never reached the point of being not bootable.

[themoonisacheese@sh.itjust.works]

My latest project runs on a VM I use vscode's ssh editing feature on. I edit the only copy of the file in existence (I have made no backup and there is no version control) and then I restart the systems service.

So what if I mess it up? Big deal. The discord bot goes down for a few minutes and I fix it.

Same goes for the machine configs. Ideally the machines are stable, the critical ones get backups, and if they aren't stable then I suppose the best way to fix it would be in prod ( my VMs run debian, they're stable).

[notabot@lemm.ee]

I manage all my homelab infra stuff via ansible and run services via kubenetes. All the ansible playbooks are in git, so I can roll back if I screw something up, and I test it on a sacrificial VM first when I can. Running services in kubenetes means I can spin up new instances and test them before putting them live.

Working like that makes it all a lot more relaxing as I can be confident in my changes, and back them out if I still get it wrong.

[ambitiousslab@lemmy.ml]

For services only I depend on, I have production-only. Since I can only inflict damage on myself, and can often work around problems.

For the XMPP server my friends and family also depend on, I have a dedicated nonprod VPS. My services are driven by ansible playbooks, so I'll tweak the playbook with whatever change I want to make works in nonprod, before running the same playbook against prod.

Whenever there's a new Debian Stable release, I'll rebuild the servers completely, to try and prevent "drift" between the nonprod and prod versions (not that I change things often enough for this to become a big problem). This is also the big test of my backups, which so far haven't been needed in a "real" emergency

[humanperson@sh.itjust.works]

Eh, I sometimes spin up a temporary docker container for some nonsense on a separate computer. I usually just go for it after checking no one is on and backing up necessary data.

[morpheus17pro@lemmy.ml]

In my case, yes. My setup is managed using Ansible playbooks, so I have a dev inventory and a playbook that spins up a virtualized environment that mimics (as much as possible, as there are a few details that cannot be fully replicated) my home lab.

That way, I usually prepare my new setups on dev, and then deploy on my pro setup and test with the few aspects I cannot reproduce in dev.

Finally, I have everything backed by a (private) git repo.

[r0ertel@lemmy.world]

After breaking "prod" many times, I have a Dev (local machine), Test (small VM) and Prod (big VM). My test is just less RAM and space and I need to spin down certain K8s things to spin up others, but it's a close mirror of Prod, just less.

[sugar_in_your_tea@sh.itjust.works]

I test stuff on my laptop or desktop and then push it to my NAS. Everything is also containerized and snapshotted, so risk of breaking anything is pretty small.

[invertedparallax@lemm.ee]

In my job? Yes.

At home? God no.

I make sure I can recover data when things go wrong, but otherwise my recovery path is redeploying quickly.