After almost half a century, I'm still doing it...
-
This is a server I was setting up. It's not doing anything useful at all at the moment, hence the lax work practice. The only reason I drove back to work is because it's needed tomorrow and I wanted to finish setting it up tonite.
-
Until they have to troubleshoot the console server ..
-
then setup a super console server. lol
-
Did this once on a router in a datacenter that was a flight away. Have remembered to set the reboot in future command since. As I typed the fatal command I remember part of my brain screaming not to hit enter as my finger approached the keyboard.
️ -
Have remembered to set the reboot in future command since
That's not a bad idea actually. I'll have to reuse that one.
Thanks! -
You’d think you’d learn from your mistakes
Yes, that what you'd think. And then you'll sit with a blank terminal once again when you did some trivial mistake yet again.
A friend of mine developed a habit (working on a decent sized ISP 20+ years ago) to set up a scheduled reboot for everything in 30 minutes no matter what you're going to do. The hardware back then (I think it was mostly cisco) had a 'running conrfig' and 'stored config' which were two separate instances. Log in, set up scheduled reboot, do whatever you're planning to do and if you mess up and lock yourself out the system will restore to previous config in a while and then you can avoid the previous mistake. Rinse and repeat.
And, personally, I think that's the one of the best ways to differentiate actual professionals from 'move fast and break things' group. Once you've locked yourself out of the system literally half way across the globe too many times you'll eventually learn to think about the next step and failovers. I'm not that much of a network guy, but I have shot myself in the foot enough that whenever there's dd, mkfs or something similar on the root shell I automatically pause for a second to confirm the command before hitting enter.
And while you gain experience you also know how to avoid the pitfalls, the more important part (at least for myself) is to think ahead. The constant mindset of thinking about processes, connectivity, what you can actually do if you fuck up and so on becomes a part of your workflow. Accidents will happen, no matter how much experience you have. The really good admins just know that something will go wrong at some point in the process and build stuff to guarantee that when you fuck things up you still have availability to fix it instead of calling someone 6 timezones away in the middle of the night to clean up your mess.
-
We've all been there. If you do this stuff for a living, you've done that way more than once.
-
Every network engineer must lock themselves out of a node at some point, it is a rite of passage.
-
It's console servers all the way down (up?)
-
If it makes you feel any better, I did something just as infuriating a few years ago.
I had set up my home media server, and had finally moved it to my garage with just a power cable and ethernet cable plugged in. Everything was working perfectly, but I needed to check something with the network settings. Being quite new to Linux, I used a remote desktop tool to log in and do everything through a gui.
I accidentally clicked the wrong item in the menu and disconnected the network. I only had a spare ps/2 keyboard and mouse, and as the server was an old computer, it would crash if I plugged a ps/2 device in while it was running*.
The remote desktop stayed open but frozen, mocking me for my obvious mistake and lack of planning, with the remote mouse icon stuck in place on the disconnect menu.
*I can't remember if that was a ps/2 thing, or something specific to my server, but I didn't want to risk it
-
I formated an OS drive by mistake last night, thought it was my flash drive...
-
I have once actually used a console server console server to troubleshoot a misbehaving console server.
-
Almost did the same last night on a device that has its internal drive (flash) mounted as mmc and the USB drive was sda
-
and you make each one geographically closer than the previous one until there's one right next to you. lol
-
i once worked at a place that had something like this and; it sounds silly; but i got a live demonstration that it was the smartest thing ever.
-
Good to know that in another 30 years, I will still be doing the dumb shit I've been doing for the last 20.
-
Gotcha. I disagree with your methodology of not being careful about non-prod systems. You stated you forgot it was remote. What happens the next time you do that but forget it's prod or mix up terminals?
-
That entire scenario scares me lol
-
Old hardware used to get really upset before plug and play became common. I remember I was playing some old racing game with a joystick on a win95 box, and accidentally pulled the connector out, lost my entire game because the system flipped out.
-
I was on-call and half awake when I got paged about a cache server’s memcached being down for the third time that night. They’d all start to go down like dominoes if you weren’t fast enough at restarting the service, which could overwhelm the database and messaging tiers (baaaaad news to say the least). Two more had their daemon shut the bed while I was examining it. Often it was best to just lick it on all of them to rebalance things. It was… not a great design.
So I wrote a quick loop to ssh in and restart the service on each box in the tier to refresh them all just in case and hopefully stop the incessant pages. Well. In my bleary eyed state I set reboot in the variable instead of restart. Took out the whole cache tier (50+) and the web site. First and only time I did that but that definitely woke me up.
