Jeremy’s employer, SwissMedia, were upgrading their proprietary CMS to run on new, shiny, PHP5. They planned for bumps in the road, but assembled a rugged upgrade plan with a steel chassis. When the time came to upgrade their largest client, French-Haitian News, Jeremy was behind the wheel.

The first step in the plan was for Jeremy to take a copy of their production database that he could experiment with and work out the kinks. He would then prove it worked with the PHP5 application, and get the stamp of approval to go to production. SwissMedia outsourced their data storage, so he contacted Sebastien at Datamaniaks to handle that part.

Somewhere between getting the data and making it work with the PHP5 application, Jeremy committed the dreaded “forgotten WHERE clause” boner. His local copy of the French-Haitian News DB became unusable. He immediately reached out to Sebastien to help remedy the situation.

“Hey, Sebastien,” his email started, “I’m sure I’ll have a good laugh about this later, but I totally trashed the test DB. Could you send me today’s 2AM backup when you get a chance?” He laid out the specific database backup he wanted, and where it needed to be delivered. Jeremy then skipped out for an early lunch while waiting for Sebastien to give him the goods.

Upon his return from lunch, Jeremy didn’t find any “goods”, but plenty of bads. The website director from the French-Haitian News had stuffed Jeremy’s voice-mail inbox full of angry messages. “Our content is out of date! We’ve lost ALL OF TODAY’S ARTICLES!” Jeremy pulled up the F-HN site and confirmed the issue- the last article had a timestamp of 0150.

Jeremy called Le Directeur and assured him they were looking into the problem. All he got in return was audible venom. Jeremy managed to tame the cobra by suggesting they start working on re-uploading all the content they had posted since 1:50AM. In the meantime, Jeremy could get to the root of the problem.

Jeremy opened up Outlook, ready to fire off an email to Datamaniaks with a big red exclamation point. Waiting for him was an email from Sebastian: “No problem, Jeremy! You just need to be more careful. LOL! I’ve restored the production database from the 2AM backup so you should be all set.”

The thud from Jeremy’s jaw hitting his desk could be heard across the SwissMedia office. Jeremy engaged his Caps Lock key and replied, “THAT IS NOT WHAT I AKSED FOR SEBASTIEN. I WANTED YOU TO SEND ME A COPY OF THE PRODUCTION BACKUP, NOT RESTORE FROM BACKUP! My customer is uploading the articles again, but if you send me the production database that I ORIGINALLY ASKED FOR, I can probably fix it faster than they can, then restore that version to production. Get me the database IMMEDIATELY and I’ll work things out with the client.”

Five agonizing minutes later, Sebastien replied, “Sorry for the confusion… I’m starting the production DB upload now!” As soon as the SQL dump was done transferring, Jeremy’s phone began to ring. It was Le Directeur again, and he had discovered an entirely new dimension of pissed off. “I NO LONGER HAVE A NEWS WEBSITE YOU *CONNARD! [long string of French obscenities redacted]” Jeremy went to the F-HN website, only to get a database error staring back at him.

After the stunned silence passed, Jeremy pulled up the directory where Sebastien had supposedly uploaded the database backup. It was not a backup. There sat the actual database file. Sebastien had apparently cut and pasted it to SwissMedia, removing it from where it was supposed to be, and leaving the F-HN website dead.

Jeremy let out his own extensive string of “pardon my French” obscenities. It was a long road to cleaning up the disaster. The aftermath saw much anticipation- there were two contracts everyone wanted to expire: the one SwissMedia had with Datamaniaks and the one French-Haitian News had with SwissMedia.

Image sources: Database, Trash

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!