Victor’s first task of the day was basic maintenance, applying OS patches and security updates to the staging servers under his wing.  If nothing blew up after a day, he’d proceed to their production counterparts. The routine for each staging server update began with a blanket email notification to developers. If no one objected after 15 minutes, he’d install the patches and reboot.

Light from the rising sun had just started to glint off the herd of skyscrapers outside Victor’s window. That early in the morning, most of the people who might whine about needing five more minutes were crawling along the interstate or idling in an eight Hummer-deep Starbucks drive-thru. His first several updates completed without a hitch. Victor grabbed a quick cup of breakroom coffee during one of the reboot windows. Any time things went this smoothly, it sent his Murphy-sense tingling, but he forced himself to walk slowly and bask in the early-day calm.

Serenity lasted until he got back to his desk. Victor found a flood of new messages in his inbox, more than half with red exclamation points and misspelled caps-lock subject lines. He didn’t know any of the senders. A skim of the subject lines revealed something called “TTP” had gone down.

Victor was painfully familiar with the applications in his care, and TTP wasn’t one of them. He brought up a command prompt and assessed the health of each production server, just to be sure. They were all responding; all ports he knew to be listening were accounted for. He figured he’d been copied on these emails only because someone, somewhere had vaguely remembered he worked in IT.

Victor sighed, sat back, and waited for the latest rebooting server to come back up. His desk phone rang a moment later- his boss. This early? He always beat his boss to the office by an hour or more. Victor picked up.

“The TTP outage- where do we stand?” she asked. “R&D out on the east coast is breathing down my neck about lost revenue.”

Victor frowned. “I’ve been seeing all those emails, but as near as I can tell, that’s not our app.”

She paused. “They say it went live on our servers yesterday.”

“What? I haven’t promoted anything to production in weeks.” Victor switched from receiver to headset, and brought up the company ticket tracking system. “Do you have the request number for their promote?”

“No.  Well, are any of our servers having issues right now?”

“No.”

Another pause. “Hang on, I’ve got a URL for the application in this email chain. Can you figure out what server it’s running on?”

Victor tamped down annoyance. At least a quick ping would soon absolve him. He typed in the command, squinted at the response… and braced his arms to prevent his head from slamming onto his keyboard.

“Boss? That’s a staging server,” he said. “It’s rebooting right now. I sent out a warning email before I bounced it, but I never heard anything.”

“OK.” Relief and confusion tinged her voice in equal measure. “OK, so that server will be back up soon. Good. Now, how did a production app get deployed to staging in the first place?” She wasn’t blaming Victor, just baffled by the situation.

“I have no idea,” Victor replied.

“Track down the promote request and any other info you can, and meet me in my office at 9:00,” she said. “I’ll round up the TTP project team so we can figure out where the wires got crossed.”

Victor scoured old emails, and did his best with his ticketing system’s terrible search functionality, but couldn’t find anything related to TTP. At 9:00, he feared walking into a near-empty office with no leads, but his boss had managed to corral the entire TTP project team onto the same conference line, and demanded answers of them first. She got to the bottom of the mystery quickly enough.

“I went through the IT help desk to promote the application,” the project manager explained without shame. “They don’t require me to fill out half a dozen authorizations.”

“One authorization,” Victor’s boss snapped. “It exists for a reason. You realize this business-critical app is not actually in production as a result?”

“I promised R&D a go-live of April 1st. Development and UAT didn’t finish until March 29th. There was no way you’d get it promoted that quickly,” the PM returned. “I’m glad everything’s up and running, but you’d better make sure it stays that way.”

It took another six weeks of such wrangling before the project manager filed the proper paperwork to get TTP hosted in production. For the entire span, Victor had to treat the TTP staging server as production-esque, and watch his email like a hawk to deny reboot requests from developers.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!