"John, it's about time I showed you how to do a production install," Dave said.

John had only been at the job for a few weeks, and was still learning the ins-and-outs of their shop. It was a small team, with the stereotypical alpha-geek at the top, people like Dave in the middle, and John at the bottom. They lived to support a complex pile of applications that all existed to extend or manage their flagship application.

Said flagship was a .NET web app with hundreds of external corporate customers that considered it "mission critical". It was still in .NET 1.1, but the upgrade was coming soon. It was still running on SQL Server 2000, but again, the upgrade was coming soon. That the flagship was a little rickety didn't matter; the application was a "big deal", and that was what the business cared about.

"So, you know the Enterprise Manager," Dave said as he pulled the application up and logged into the test database. "I'm going to export a CREATE script for all of the stored procedures off the test DB." With that done, he explained the next steps as he executed them: log onto production, navigate down the tree to where the stored procedures were, delete that entire branch of objects without mercy, and run the script to recreate them.

While they waited for the delete step to complete, John asked, "Does it usually take this long?"

"No, not usually," Dave admitted. He frowned and the screen and waited a few minutes longer before adding, "I hope it isn't hung… I always worry about just killing something like this. I'm afraid I'll lose data in the database." The task finally completed, and Dave ran the script to create the procedures.

It failed. The procedures, it seemed, already existed. Dave expanded the branch and confirmed that they were still there. "Then what did I delete?" he wondered.

A frantic call from the user support reps answered that question: the tables. Dave had selected the wrong branch and deleted gigabytes of customer generated data. Stunned by the immensity of his screw up, Dave stared at the screen, like it was the oncoming headlamps of a semi. "Should… should we start restoring from backup?" John asked.

"Backup?" Dave said the word, but it held no immediate meaning. "Backup? Backup!" The semi sailed by, and Dave relaxed. The day was saved! They could restore from backup! Dave started hunting down the most recent backup so that they could get the restore underway ASAP.

Unfortunately for Dave, the most recent backup was months old. The backup maintenance task, also Dave's responsibility, had fallen out of the scheduler, and no one had ever confirmed that it was running. The next day, John helped Dave's replacement, Phil, reverse engineer as much data as they could off of other servers and databases. It took weeks to complete. The company lost hundreds of thousands of dollars from the downtime alone, breached several contracts, lost several major customers, and a significant chunk of their reputation.

It wasn't all a loss. John learned a valuable lesson about how not to do a production install.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!