After almost half a century, I'm still doing it...
-
Good to know that in another 30 years, I will still be doing the dumb shit I've been doing for the last 20.
-
Gotcha. I disagree with your methodology of not being careful about non-prod systems. You stated you forgot it was remote. What happens the next time you do that but forget it's prod or mix up terminals?
-
That entire scenario scares me lol
-
Old hardware used to get really upset before plug and play became common. I remember I was playing some old racing game with a joystick on a win95 box, and accidentally pulled the connector out, lost my entire game because the system flipped out.
-
I was on-call and half awake when I got paged about a cache server’s memcached being down for the third time that night. They’d all start to go down like dominoes if you weren’t fast enough at restarting the service, which could overwhelm the database and messaging tiers (baaaaad news to say the least). Two more had their daemon shut the bed while I was examining it. Often it was best to just lick it on all of them to rebalance things. It was… not a great design.
So I wrote a quick loop to ssh in and restart the service on each box in the tier to refresh them all just in case and hopefully stop the incessant pages. Well. In my bleary eyed state I set reboot in the variable instead of restart. Took out the whole cache tier (50+) and the web site. First and only time I did that but that definitely woke me up.
-
If I understand, this is less a complaint about how UNIX works and more a story about the consequences of careless mistakes.
This is why we have KVMs I guess. Though not every server has one of those.
-
At $DAYJOB, we're currently setting up basically a way to bridge an interface over the internet, so it transports everything that enters on an interface across the aether. Well, and you already guessed it, I accidentally configured it for
eth0and couldn't SSH in anymore.Where it becomes fun, is that I actually was at work. I was setting it up on two raspis, which were connected to a router, everything placed right next to me. So, I figured, I'd just hook up another Ethernet cable, pick out the IP from the router's management interface and SSH in that way.
Except I couldn't reach the management interface anymore. Nothing in that network would respond.Eventually, I saw that the router's activity lights were blinking like Christmas decoration. I'm guessing, I had built a loop and therefore something akin to a broadcast storm was overloading the router.
Thankfully, the solution was then relatively straightforward, in that I had to unplug one of the raspis, SSH in via the second port, nuke our configuration and then repeat for the other raspi. -
Nah, you only need two, each connected to the other. Use one to work on the other.
-
I don't buy servers without iDRAC enterprise licenses. It's too damn useful.
-
I've been trying to find a network capable KVM for home use. They're all pretty expensive or lacking functionality. I don't actually need one or I'd pull the trigger, but I sure have been tempted.
-
A decade and change ago, in a past life, I was tasked with switching SELinux to permissive mode on the majority of systems on our network (multiple hundreds, or we might have gotten above one thousand at that point, I don't recall exactly). This was to be done using Puppet. A large number of the systems, including most of our servers, had already been manually switched to permissive but it wasn't being enforced globally.
Unfortunately, at that point I was pretty familiar with Puppet but had only worked with SELinux a very few times. I did not correctly understand the syntax of the config file or
setenforceand set the mode to ... Something incorrect. SELinux interpreted whatever that was as enforcing mode. I didn't realize what I had done wrong until we started getting alerts from throughout the network. Then I just about had a panic attack when I couldn't login to the systems and suddenly understood the problem.Fortunately, it's necessary to reboot a system to switch SELinux from disabled to any other mode, so most customer facing systems were not impacted. Even more fortunately, this was done on a holiday, so very few customers were there to be inconvenienced by the servers becoming inaccessible. Even more fortunately, while I was unable to access the systems that were now in enforcing mode, the Puppet agent was apparently still running ... So I reversed my change in the manifest and, within half an hour, things were back to normal (after some service restarts and such).
When I finally did correctly make the change, I made sure to quintuple check the syntax and not rush through the testing process.
-
So that's why we have mobile phones
-
I assume you've never made any mistakes, ever. What an arrogant attitude.
-
I have made mistakes. I will make more mistakes in the future. I will not repeat disruptive freshmen mistakes like the one described here.
-
A few months ago I accidentally dd'd ~3GiB to the beginning of one of the drives in a 4 drive array... That was fun to rebuild.
-
It's not Unix, it's you.
-
I had a remote relay box: 8 channels of power control, so I could at least power cycle machines from remote when all else failed.
I actually ended up not using it much at all, it was a nice security blanket, but the last time I decided that I wanted to power cycle something was about 6 years ago, and at that time I realized it had been over 3 years since I had previously used it, and that usage was more of a "let's make sure this thing is working like I think it should" test.
-
For clarity, I have done it myself - plenty, but not just on Unix boxes.
-
Your 4 drive raid5 array, right?
Right?!
-
so long as you're mobile, any phone can become a mobile phone. lol