Postby Okaros » 25 Jul 2017, 17:07:15
Since you asked...
The new global sysadmin/networking manager where I work is learning a set of profound lessons this week.
Angry Story Time!
A week ago:
Networking Lead: This border-switch replacement will be only be 2 minutes of downtime, tops! Customer Support has ok'd it happening without a maintenance window if it's under 2 minutes!
Me, to Manager: Swapping out the switch doesn't fix the connectivity problem, it just lets you actually manage the switch to *start* diagnosing the problem. You'll need a multi-hour maintenance window, which is fine. Just schedule it, the changes I have waiting on this fix can wait for you to do it right.
Manager, to me and Customer Support Lead: Okaros, you're right. I'll talk to the Networking Lead and we'll get a Maint Window scheduled.
This morning:
Networking Lead: Oops, we screwed up and caused over an hour of unplanned, unannounced production downtime less than a month after a total datacenter failure that lasted 18 hours!
Customer Support: *fiery rage inferno as major customers go ballistic*
Me: *picard facepalm* Wait, wtf? This work was cancelled and the last notice that was sent said you were going to schedule a maintenance window!
Networking Lead: Manager approved the work!
Manager: Yeah, uh, I did approve that. Um. Oops. I also dropped the ball on communicating that we were still going to do the work to anyone at all other than the Networking Lead.
Me: So things are fixed now, at least?
Manager + Networking Lead: No, we had to roll back the changes. We're exactly where we were before.
Me: ...
Honestly, it's not even the Manager approving the change over my explicit objections that's driving me berserk (by trial and error he''ll eventually figure out that I am right 100% of the time). It's the fact that there was zero communication to anyone about it. He should know better (he's new to the company, but not new to managing production IT resources). It's that they left absolutely everyone thinking it wasn't happening. Manager isn't new to managing production IT resources, so there's just zero excuse.
Manager and I both report directly to the Director of Cloud Services, and this was visible all the way up to the CEO. One of us is spending the rest of the week being chewed to pieces by multiple Directors and C-Levels from across the company and has earned a serious black mark on their record. Hint: It's not me.
I kind of assume Okaros is out to backstab me until proven otherwise. - DOM