The ICSP compliance application was a dull piece of software. Instead of managing Active Directory with one of the many off-the-shelf packages, this was built in-house, and helped the sysadmins manage user accounts, permissions and access. Despite the fact that this application had no right to exist, it did, and it was tied into so many IT business processes that ICSP was marked a “critical” application.

Kelly spent too much time with her hands deep in the entrails of this ugly creature. Since she “owned” it, she was the recipient of a panicked email: “ICSP is down! What did you do?” She had changed a few things is code, recently, but nothing had been released to test, let alone production. Quickly, she skimmed the email chain.

At the bottom was an exchange between end users; Kelly didn’t recognize their names. That bubbled up to the Service Desk, which saw ICSP and escalated it to IT Security Services, the team which used the application. There was a pile of confusion there, as the team had no idea what the issue was, so they escalated it up to their manager, than up again, until it finally reached Mr. Dorn, the Chief Security Officer. Finally, that raised the issue up high enough that it could drop back down the software development side of the org-chart- it rolled down through her management chain and landed on her desk, a big smelly pile of nothing she wanted to deal with.

At no point in the email chain did anyone discuss error messages, or symptoms beyond, “I saw a message that said ICSP is down.” As Kelly turned to check what was actually happening in production, her boss appeared at her cube. “Hey, did you see about ICSP?”

“Yes, I’m…”

“I’ll need you to jump right on that.”

“Yes, I’m…”

“It’s super high-visibility, and we need to keep Mr. Dorn happy.”

“Yes, I’m…”

“Try reaching out to someone from ITSS to see if they can help.”

“Well, I don’t even know what’s wrong…”

Her boss continued to blather, and Kelly did her best to tune him out. It didn’t take long to confirm that ICSP was up and running happily. She emailed the original reporting user, asking for a screenshot of the message. They replied with an Excel spreadsheet attachment, which contained a pasted “print screen” of their entire desktop.

The outage message wasn’t from ICSP, nor was it on anything to do with IT Security Services- it was on the Global Financial Services SharePoint page. They were responsible for finance and things like Sarbanes-Oxley compliance, but they weren’t users of the ICSP application. Even so, in bright red letters, they warned the users:

Do NOT use ICSP. The site is currently undergoing maintenance. Contact Janine with any questions.

Kelly emailed Janine, but got an “out of office” message in reply. Beside the error message was a link labeled “ICSP Application”. Kelly clicked it, expecting to see the hideous, retina-scarring awfulness of her ICSP app. Instead, she saw a slightly less hideous interface to a completely different application. The only thing it had in common with her ICSP was the title. “ICSP” stood at the top of the screen in the best formatting an unstyled <H1> had to offer. The URL, however, called it “soxonline”. Also, the application was quite clearly up.

Kelly knew nothing about this new ICSP application, aside from its name, and it certainly wasn’t her responsibility, but the still-growing email chain of panicked and confused management had started reciting the politician’s syllogism. Knowing it was probably a mistake, Kelly took the initiative and poked around in the code for the other ICSP application.

This other application was a document management tool, so that various groups could fill out an “ICSP document” to meet SOX compliance. Her ICSP application was actually so named because it was IT Security’s attempt to meet the requirements they set out in their ICSP document. The document application was globally accessible, which let Kelly confirm that this other application was still up, too. She explained the confusion in an email, saying, “I’m not sure why the error message is there, but the application is working.”

“Well,” Mr. Dorn replied, “that doesn’t solve anything. Why is that error message there? I’m CCing this to the Performance and Resolution team, so they can take a look.”

The email chain exploded again, as an entirely new IT team swarmed to the problem. P&R spun up new instances of their monitoring tools on both ICSP servers. Their managers, wanting to make sure that everyone knew they were taking this issue seriously, started blasting out half-hourly updates, and then started assigning action items to Operations. Operations kicked it over to the SharePoint team, where the issue sat and died; there wasn’t a SharePoint team, just a few operations folks that were tagged as knowing something about SharePoint.

Kelly’s inbox continued to explode, so she reached over to kill her email client. Her fingers were hovering over ALT+F4 when one fresh email appeared, from Janine- the user mentioned in the error message.

I’m out of the office for the next six weeks, and I didn’t want anyone to use ICSP when I was gone. The easiest way to do that was to update the SharePoint site and tell people the application was undergoing maintenance.
Thanks,
Janine

Kelly forwarded that to the panic-chain of emails, and closed her mail client for the day.
Image from “Star Trek: The Next Generation”

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!