An Easy Friday Job
Ever have one of those days? We visualize success pertaining to a particular project and yesterday was no different. The job: Pick up a customer router from one location, drive it to a different location, and make the configuration changes necessary for working internet access. I’m thinking 30 minutes, tops. The network infrastructure at the new location is ready to go. The customer is responsible for moving all other equipment including the server. The kind of job I look forward to as a network professional.
The Filthy Unknown
I arrive on site to the first location and breathe a sigh of relief that the old 05 Focus Wagon’s muffler is still attached. That’s a story for another day. I’m promptly presented with a slightly larger project than I was initially tasked with; one involving the original plan of picking up the old router but with some additional extras. Included in those extras: Loading up and moving the Server, Switches, Backup Drives, Keyboards, Monitors, and Mice. Speaking of mice, I found a few dead ones while gathering up all of the patch cables scattered throughout the building. The place was nasty! It appeared to have never ever once been cleaned and I felt – dirty. Luckily I had a near full bottle of hand sanitizer in the car.
Nearly an hour into this project, I get everything loaded up and I finally leave that dirt fortress. I arrive at the new location about 15 minutes later; the hand sanitizer bottle now down to half capacity. I spend 15 minutes carrying all of the previously mentioned items to their final destination. I get the switch all patched up and I’m ready to fire up the server. Hmm, what’s this? Red Heart Beat light? Umm, ok. Fire it up! HP Proliant ML350 fan speeds mimic the sound of a small jet engine. I see errors on the screen iLO FW error. Storage Controller Battery 1 Failure. You have got to be kidding me! I power down and crack her open. No loose cables. I re-seat every thing and button it back up. Fingers crossed…
Panic Sets In
Talk about a SNAFU. I’ve been in this situation before but it’s not so much I’m the one that is panicking. The owner has that concerned look on his face again. You see, he’s been in this situation before, too. The old filth hole we left earlier must have had a bad electrical ground or something. He has lost an entire phone system twice and I recall replacing a couple of T1 modules in the Cisco. My best educated guess is either when disconnecting the power for the server from the old location or possibly when connecting it in the new location, something was zapped on the main board. I put in the dreaded call to HP. They inform me the server is covered until 2019. The recommendation is an iLO firmware update to resolve the jet engine fan issue and then we take it from there. Further adding to the amount of frustration of everything wrong with this day, the ISP technician has not been on site yet so I have no internet access to work with. I’m dead in the water.
Who Wants It
Six hours pass by. I pack up the server and drive it back to the office. I broadcast an email for anyone interested in a potential overtime opportunity as this one is most likely going to run through the weekend. Being a full time family man and student along with a bucket full of other life stuff, I no longer work outside of normal business hours. Luckily everyone else in the office loves the OT rates and it wasn’t long before every tech in the building was inbound to troubleshoot. I left the office cursing the technological Gods. I always feel bad for a customer who is experiencing a catastrophic outage. I’m comforted by the thought of my experienced coworkers for promptly stepping up to this issue late on a Friday afternoon.