Not too long ago, The Powers That Be in the emerging eastern-European country of Latveria (as I’ll call it) decided that the time had come for a massive, central monitoring system that would be used to ensure the country’s security. SENTINEL, as the system would be called, involved data exchange between virtually every governmental agency, airports, financial institutions, transit systems, and so forth, all for the purpose of being able to track people and the money they spend. After well over a year of negotiations, The Powers That Be selected Christian B’s company to design SENTINEL’s enterprise architecture.

Over the next year or so, Christian got accustomed to the weekly Latverian commute: wake up at 4:00 AM on Monday, depart from Paris via airplane, arrive in Hassenstadt (Latveria’s capitol) several hours later, work five twelve-to-fifteen hour days, catch a plane back home, and sleep until noon on Saturday. He also got very accustomed with Latverian politics: The Powers That Be, especially those from the Ministry of Defense, were personally involved with almost every decision at every level. And they didn’t take criticism very well. One network engineer was fired on the spot (and some suspect, later executed) for disagreeing with a Latverian director.

Despite all the bureaucracy, the number of parties involved, the complexity of the decision-making process, and the constant design changes, SENTINEL was actually delivered on time and nearly 70% feature complete. All the hard work that Christian and his team put in – along with the threat of capital punishment should the project fail – helped out quite a bit. The missing features – left out mostly as a result of external systems – would be delivered months later in SENTINEL 2.0. And fortunately, Christian would have no part in this second project.

Upon returning to the office in Paris following a month-long holiday, Christian’s boss told him that he’d need to pack his bags and head to Hassenstadt right away to help with an audit project. Although Christian argued that there were several others more qualified to do audit work, especially those who didn’t design the system, his boss was insistent. Apparently, the Director of Information Systems personally requested Christian’s presence.

Shortly after arriving in Hassenstadt, Christian jumped in to the audit team meeting. They were discussing all the fun they’d be having over the next months: architecture review, code review, deployment review, functional test review, database performance review, incident simulation, and so on, all with Latverian oversight. Christian couldn’t wait.

On a coffee break, Christian ran into a developer he had worked with on the project. After the standard small talk, Victor (as I’ll call him) asked Christian a rather odd question: “Have you heard about my laptop?”

“Sorry,” Christian replied, “I haven’t?” He half-expected Victor to tell him that some Latverian general smashed it to pieces after learning about some system limitation. It wouldn’t have been the first time.

“Ha,” Victor chuckled, “you didn’t hear?! Well, OK, you just have to see this in person.”

Christian followed Victor down the stairs towards the SENTINEL data center. It took up an entire sublevel in the massive Ministry of Defense building they were all meeting in, and housed tens of millions of dollars worth of servers and network equipment. The room stayed a cool 55° and was lined with racks of high-availability hardware, and more racks of backup high-availability hardware. It was exactly how Christian had designed it.

However, there was something out of place. A single chair sat in the middle of room and, on this chair, sat a beat-up laptop with a few wires connected to it from the surrounding racks. It was like some sort of totem, completely contrasting the otherwise perfectly clean room. The laptop’s screen displayed the Performance tab from Windows Task Manager with a CPU-usage graph almost constantly at 100%. The only other program on the taskbar was Visual Studio.

Christian was surprised and baffled. A laptop certainly was not in his architectural specification of the system!

When they left the cold room, Victor told Christian the whole story. As more and more data poured in and out of the system, and more and more people started using the system, one of the application servers crashed. And then it crashed again. And again. And again. For some reason, one of the application’s services would crash whenever it started up.

As a developer working on this part of the service, Victor was called in to fix and track down the bug. He looked at the logs and saw exactly where the problem was: some UTF-8 character string caused the service to crash. It was odd, but must have been some bug in the request processing.

He fired up the service through Visual Studio on his laptop and submitted the same data. It didn’t crash. He tried it again, and it worked. And again. And again. Nothing he could do would crash the service on his laptop. This was not good.

Victor explained the situation to his manager. They both knew that, if the crashes were noticed by the higher-ups in the Ministry of Defense, or worse, by The Powers That Be, heads would roll. And probably literally. They held a quick emergency meeting with the other development managers to get some help.

“Well,” one of the managers said, “the service runs fine on your laptop, right? Well, let’s just plug it to the production network! In the mean time, we can examine exactly why it’s failing on the application server.”

Without any other ideas, Victor and his manager left the meeting and begin the IDDTP procedure: Install, Deploy, Don’t Test, Pray. Victor’s laptop was placed on a chair and plugged-in to the network. Within moments, SENTINEL was back up and running – quite a bit slower than usual – and Victor’s laptop was working like mad.

A few days of tedious debugging later, Victor’s team discovered that the problem was in some third-party communications library. Try as they might, they could not get the application server to function properly. Victor’s laptop had just the right combination of patches, service packs, Win32 DLLs, and whatever other magic was needed to not crash the vendor’s library. And they were still in the process of fighting with the vendor to fix their library.

And that’s the reason why Victor’s laptop – an underpowered, $1,100 development computer – had been sitting between multi-million massively-available dollar hardware, courageously powering the huge enterprise application called SENTINEL. And fortunately, The Powers That Be were none the wiser.

Please let me know what you think of WTF! Take the WTF Reader Survey!
[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!