After almost half a century, I'm still doing it...
-
A decade and change ago, in a past life, I was tasked with switching SELinux to permissive mode on the majority of systems on our network (multiple hundreds, or we might have gotten above one thousand at that point, I don't recall exactly). This was to be done using Puppet. A large number of the systems, including most of our servers, had already been manually switched to permissive but it wasn't being enforced globally.
Unfortunately, at that point I was pretty familiar with Puppet but had only worked with SELinux a very few times. I did not correctly understand the syntax of the config file or
setenforceand set the mode to ... Something incorrect. SELinux interpreted whatever that was as enforcing mode. I didn't realize what I had done wrong until we started getting alerts from throughout the network. Then I just about had a panic attack when I couldn't login to the systems and suddenly understood the problem.Fortunately, it's necessary to reboot a system to switch SELinux from disabled to any other mode, so most customer facing systems were not impacted. Even more fortunately, this was done on a holiday, so very few customers were there to be inconvenienced by the servers becoming inaccessible. Even more fortunately, while I was unable to access the systems that were now in enforcing mode, the Puppet agent was apparently still running ... So I reversed my change in the manifest and, within half an hour, things were back to normal (after some service restarts and such).
When I finally did correctly make the change, I made sure to quintuple check the syntax and not rush through the testing process.
-
So that's why we have mobile phones
-
I assume you've never made any mistakes, ever. What an arrogant attitude.
-
I have made mistakes. I will make more mistakes in the future. I will not repeat disruptive freshmen mistakes like the one described here.
-
A few months ago I accidentally dd'd ~3GiB to the beginning of one of the drives in a 4 drive array... That was fun to rebuild.
-
It's not Unix, it's you.
-
I had a remote relay box: 8 channels of power control, so I could at least power cycle machines from remote when all else failed.
I actually ended up not using it much at all, it was a nice security blanket, but the last time I decided that I wanted to power cycle something was about 6 years ago, and at that time I realized it had been over 3 years since I had previously used it, and that usage was more of a "let's make sure this thing is working like I think it should" test.
-
For clarity, I have done it myself - plenty, but not just on Unix boxes.
-
Your 4 drive raid5 array, right?
Right?!
-
so long as you're mobile, any phone can become a mobile phone. lol
-
I started to DBAN (wipe) my internal drive once instead of an attached drive. That was the last time I ran DBAN on a machine with any drives of value plugged in.
-
I was scared to move the cloud for this reason. I was used to running to the server room and the KVM if things went south. If that was frozen, usually unplugging the server physically from the switch would get it calm down.
Now Amazon supports a direct console interface like KVM and you can virtually unplug virtual servers from their virtual servers too.
-
Like 3 weeks ago on my (testing) server I accidentally DD'd a Linux ISO to the first drive in my storage array (I had some kind of jank manual "LVM" bullshit I set up with odd mountpoints to act as a NAS, do not recommend), no Timeshift, no Btrfs snapshot. It gave me the kick in the pants I needed to stop trying to use a macbook air with 6 external hard drives as a server though. Also gave me the kick in the pants I needed to stop using volatile naming conventions in my fstab.
-
Not SysAdmin but about a year into my first software engineer job I was working on the live DB in SQL without using BEGIN TRAN ROLLBACK TRAN.
Suffice to say I broke the whole system my making an UPDATE without a WHERE clause. Luckily we have regular backups but it was a lot of debugging with the boss before I realised it was me.
-
not RAID10 I hope...
-
Check out JetKVM
-
This.
Do it. This saved my life on more than one occasion.
You'll think “nah, it'll be fine” and then at 11pm when your brain's fried on vending machine coffee you'll be glad that you did it.. 3 times over...
-
Lol I've locked myself out of so many random cloud and remote instances like this that now I always make a sleep chain or a kill timer with tmux/screen.
Usually like:
./risky_dumb_script.sh ; sleep 30 ; ./undo.sh
Or
./risky_dumb.script.sh
Which starts with a 30 second sleep, and:
(tmux) sleep 300 ; kill PID
-
Until you block ICMP one day and then wonder why the server keeps rebooting...
(Been there. Done it)
-
Without repeating my other comment. This approach saved my life many times
