0x4E4F, 1 year ago (edited 10 months ago)

Had this really shitty day at work, last Friday of the year… my leg hurt (still don’t know why, I think it’s the footwear, but I’m not sure), had a lot of work that day (stayed overtime, not paid BTW). My wife calls me, says to pick up some things on my way home… OK. I walk home BTW, I live 30 minutes from where I work. I buy the stuff she asks… people from work start calling me, there’s problem, they can’t access some of the storages in the network… OK, I go back to work, walking on my busted leg, plus the stuff I got from the store. I get there, everything is working fine 😒… check the backbone switches, the storages, logs, it seemed like a glitch of some sort, a sync issue most probably. I start picking up my groceries and packing up to go home, shit happens again… OK, at least I’m not gone… it’s 8PM by now. My wife calls me to ask me where I am, I say I’m at work, she hangs up 🤦… now I gotta explain why I’m still there. OK, go to the server room, the problem is gone by now, inspect the logs again, same thing… at this point, I’m this close to plugging everything out of the wall and plugging it back in again… to make sure I just yank a few cables and not every single piece of equipment in the rack (in case I decide to do this), I start inspecting how everything is connected, power cable wise. So, I pull out the backbone switches from the racks, one by one, and I start hearing this buzzing sound, like a bad electrical connection/plug… mhm, OK, I think we might be getting to the root of the problem, now let’s see which one is it. I go to the back, start checking each plug on the back… everything seems OK… OK, so it’s not the power plugs… at least not on the switch’es side. Something’s still buzzing, so I try and pinpoint the sound. Seems to be coming from the lower part of the rack… darn, have to pull everything out to get in there. It’s 10PM by now. So, I pull everything out and I notice 2 power cables not in plugs, but just connected with wires to another cable that goes into a plug. Why would they do that. Sure, they were insulated with tape, but why not just plug them in. Go in a bit further to hear if the sound is coming from there, move the cables a bit, the buzzing stops, but I hear coolers spinning down… OK, I pull myself out of there, go back to the front of the rack, sure enough, one of the backbone switches is down. As it turns out, they didn’t have enough free plugs in the rack for all of the equipment, so instead of installing a new extension cord, they just cut 1 power cable, made a Y splitter for 2 of the switches and just plugged that cable in. Basically, there is no way of getting one of the switches offline, if you yank that thing out, 2 go offline. I pull out a few new power cables, get an extension cord from my office (we have a few laying around), plug those 2 the way they should be plugged, plug in the extension cord in the first empty power plug I could find in another rack, put the equpment back in the rack, leave the rack open cuz of the extension cord going to another rack and (finally!) leave work to go home, on my busted leg, with a shit ton of groceries… it’s 11:30PM by now. I get home, get the 3rd degree for arriving so late on a Friday before NYE… I go to bed, my wife says she want’s a backrub… I told her to suck it, left the bedroom and went to sleep in the living room.