Most of us IT professionals have come to learn just how cautious one must be when upgrading third-party components. Some of us have learned the hard way, first proclaiming, pssh, big deal, it's just Service Pack 3; it couldn't possibly break our application, and then learning that, in fact, Service Pack 3 was a big deal and did break the application. Others are naturally cautious and try to make sure planning, testing, and all those other things are done prior to any upgrades.

Database upgrades can be especially risky. A new version means different query optimization, new language constructs, and, in Oracle's case, the introduction of VARCHAR3 and subsequent deprecation of VARCHAR2. This is why, when Kevin's company decided to upgrade their database from SQL Server 2000 to SQL Server 2005, they put him in charge of formulating a schedule for a six-to-twelve month upgrade plan.

The first step was to install SQL Server 2005 on the development servers, giving developers the opportunity to fix whatever broken queries they noticed. Next, SQL Server 2005 would be installed on the testing servers and an exhaustive list of test cases and regression tests would be run to make sure that every bit of functionality worked exactly as it did before. Finally, once everyone signed-off, the production servers would be upgraded over a three-day weekend and a support triage would be established to deal with any issues.

After a month or two of bugfixing and query optimization, the development team reported that they were ready to move to test. Because year end (and all the holiday and vacation time associated with it) was fast approaching, it was decided to hold off on upgrading the test servers until the second week of January.

One manager, however, had a slightly different plan. He came in over Christmas weekend and upgraded the database server used by their issue management software. This went completely unnoticed as much of the company was on vacation and the issue management software fully supported SQL Server 2005.

When Kevin returned from vacation on the day following New Year's, he noticed an email sent from the issue tracker with the subject "Test email from SQL2005 Upgrade." After logging in to the issue tracker server, he went directly to the manager to ask why the server was upgraded.

"Oh that," the manager replied, "that was because I needed practice doing an upgrade."

Kevin had no idea what that meant and asked what exactly he needed practice for.

"Well," said the manager, "so that I would feel more comfortable upgrading the rest of the production servers over this past New Year's weekend. They all went through without a hitch!"

Sure enough, all the production servers had been upgraded. And now, almost two months later, they're still fixing broken queries and crashes in their flagship web application.