Canopyflyer, 1 year ago

Over 150 Major Incidents in a single month.

Formerly, I was on the Major Incident Response team for a national insurance company. IT Security has always been in their own ivory tower in every company I’ve worked for. But this company IT Security department was about the worst case I’ve ever seen up until that time and since.

They refused to file changes, or discuss any type of change control with the rest of IT. I get that Change Management is a bitch for the most of IT, but if you want to avoid major outages, file a fucking Change record and follow the approval process. The security directors would get some hair brained idea in a meeting in the morning and assign one of their barely competent techs to implement it that afternoon. They’d bring down what ever system they were fucking with. Then my team had to spend hours, usually after business hours, figuring out why a system, which had not seen a change control in two weeks, suddenly stopped working. Would security send someone to the MI meeting? Of course not. What would happen is, we would call the IT Security response team and ask if anything changed on their end. Suddenly 20 minutes later everything was back up and running. With the MI team not doing anything. We would try to talk to security and ask what they changed. They answered “nothing” every god damn time.

They got their asses handed to them when they brought down a billing system which brought in over $10 Billion (yes with a “B”) a year and people could not pay their bills. That outage went straight to the CIO and even the CEO sat in on that call. All of the sudden there was a hard change freeze for a month and security was required to file changes in the common IT record system, which was ServiceNow at the time.

We went from 150 major outages (defined as having financial, or reputation impact to the company) in a single month to 4 or 5.

Fuck IT Security. It’s a very important part of of every IT Department, but it is almost always filled with the most narcissistic incompetent asshats of the entire industry.