Testing vs Prod

[marauding_gibberish142@lemmy.dbzer0.com]

I use testing, prod and stale. Stale is simply one version behind prod in case I see something in prod I need to roll back

[lorentz@feddit.it]

I don't have a testing environment, but essentially all my services are on docker saving their data in a directory mounted on the local filesystem. The dockerfile reads the sha version of the image from an env file. I have a shell script which:

1. Triggers a new btrfs snapshot of the volume containing everyithing
2. Pulls the new docker images and stores their hashes in the env file
3. Restarts all the containers.

if a new Docker version is broken rolling back is as simple as copying the old version in the env file and recreating the container. If data gets corrupted I can just copy the last working status from an old snaphot.

The whole os is on a btrfs volume which is snapshotted regularly, so ideally if an update fucks it up beyond recovery I can always boot from a rescue image and restore an old snapshot. But I honestly feel this is extra precaution: in years that I run debian on all my computers, it never reached the point of being not bootable.

[themoonisacheese@sh.itjust.works]

My latest project runs on a VM I use vscode's ssh editing feature on. I edit the only copy of the file in existence (I have made no backup and there is no version control) and then I restart the systems service.

So what if I mess it up? Big deal. The discord bot goes down for a few minutes and I fix it.

Same goes for the machine configs. Ideally the machines are stable, the critical ones get backups, and if they aren't stable then I suppose the best way to fix it would be in prod ( my VMs run debian, they're stable).

[notabot@lemm.ee]

I manage all my homelab infra stuff via ansible and run services via kubenetes. All the ansible playbooks are in git, so I can roll back if I screw something up, and I test it on a sacrificial VM first when I can. Running services in kubenetes means I can spin up new instances and test them before putting them live.

Working like that makes it all a lot more relaxing as I can be confident in my changes, and back them out if I still get it wrong.

[ambitiousslab@lemmy.ml]

For services only I depend on, I have production-only. Since I can only inflict damage on myself, and can often work around problems.

For the XMPP server my friends and family also depend on, I have a dedicated nonprod VPS. My services are driven by ansible playbooks, so I'll tweak the playbook with whatever change I want to make works in nonprod, before running the same playbook against prod.

Whenever there's a new Debian Stable release, I'll rebuild the servers completely, to try and prevent "drift" between the nonprod and prod versions (not that I change things often enough for this to become a big problem). This is also the big test of my backups, which so far haven't been needed in a "real" emergency

[humanperson@sh.itjust.works]

Eh, I sometimes spin up a temporary docker container for some nonsense on a separate computer. I usually just go for it after checking no one is on and backing up necessary data.

[morpheus17pro@lemmy.ml]

In my case, yes. My setup is managed using Ansible playbooks, so I have a dev inventory and a playbook that spins up a virtualized environment that mimics (as much as possible, as there are a few details that cannot be fully replicated) my home lab.

That way, I usually prepare my new setups on dev, and then deploy on my pro setup and test with the few aspects I cannot reproduce in dev.

Finally, I have everything backed by a (private) git repo.

[r0ertel@lemmy.world]

After breaking "prod" many times, I have a Dev (local machine), Test (small VM) and Prod (big VM). My test is just less RAM and space and I need to spin down certain K8s things to spin up others, but it's a close mirror of Prod, just less.

[sugar_in_your_tea@sh.itjust.works]

I test stuff on my laptop or desktop and then push it to my NAS. Everything is also containerized and snapshotted, so risk of breaking anything is pretty small.

[invertedparallax@lemm.ee]

In my job? Yes.

At home? God no.

I make sure I can recover data when things go wrong, but otherwise my recovery path is redeploying quickly.