QA, UAT, and performance tests passed. John received authorization to promote his first major release to production. He checked in the code, and nothing exciting happened. No cake or streamers fell from the ceiling, but no errors or warnings bleated out at him. None. Given the complexity of the product, John’s success was the IT-world equivalent of winning both showcases on “The Price Is Right”.

John sent out emails to all the relevant stakeholders, and basked in the glow of a major project finally being done. The good feeling lasted for less than half an hour.

“The production servers are completely bogged down!” His boss clung to the threshold of his cube, breathless from the run over. “What the heck’s going on?”

Orders, orders everywhere. Tens of thousands of orders flooded into the production system. A glance at the log files explained their origins: the performance testing environment.

A horrible dread crept over John’s skin. “I must have left in some flags, or some settings- it’s pulling data from PerfTest. What should I do? Should I roll it back?” he asked.

“Yes, but get an OK from the system admin first,” his boss said.

His upcoming weekend, and potentially his career, were on the line. John emailed some attachments to the system admin on duty, then ran to the admin’s desk as though trying to beat the messages there.

The system admin had different ideas. “No need to roll the whole release back. Looks like all we need to tweak is this iBATIS query.” He tapped at his keyboard like a hacker in a bad movie. “See? Easy. Get this patch in before anything bad happens. Well, anything worse.”

Weak with relief, John filed the emergency production change request with QA and release management. Once QA signed off, it would go into production without anything more from John. Everyone went home for the weekend, confident they’d done all they could.

To John’s surprise, the production-critical patch was still sitting in QA’s queue on Monday. And on Tuesday, Wednesday…

John didn’t need to inform the project lead. “I’ve been getting calls all week from people in Shipping. They’re sick of manually canceling those bad orders,” she said while popping several headache tablets. Her next action was to call the product’s main QA point of contact, placing him on speakerphone. “What’s the holdup?” she asked.

“We don’t have the financing to handle our workload,” the QA worker complained.

The project lead told her boss. John told his boss. They contacted the QA manager. Thus was the battle joined. The managers armed themselves with contact lists and SLAs. Conference calls and blamestorming sessions were the proving grounds over which these mid-tier titans clashed. With career interests and bottom lines dangling at the precipice, each vied to do as little as possible and spend the least amount of money while claiming the most credit. John’s urgent patch, a whopping eight-character code change, languished in limbo.

Nine months later, casualties stank up the battlefield. 1,000,000 bogus “orders” sent from the performance environment; $6,000,000 in losses for the company due to erroneous shipments, erroneously canceled real shipments, and angry customers. The only good thing was that by the time the damage was accounted, everyone had forgotten who released the bad code to production.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!