Photo Credit: 'ntenny' @ Flickr Having worked with several companies as a systems administration consultant, Massimo had learned that one cliché is, in fact, true: the bigger they are, the harder their bureaucracies are to navigate*. When his employer sent him to work with their biggest client – a large government agency – he was prepared to deal with endless TPS Reports, Process for Application Implementation and Navigation documents, and Form Request Forms. But there was one thing he did not expect: that little asterisk at the end of the cliché.

You see, the “bigger they are” rule only works for reasonably-sized values of “big”. Once an organization grows too big – say, large-government-agency big – strange things tend to happen. The entire bureaucratic structure can collapse on itself, creating an überbureaucracy (i.e., an Ouroboros-like bureaucracy that can serve only itself) surrounded by satellite units that somehow work together to solve the organizational goal. It’s pure chaos. And not the good, entrepreneurial/start-up kind of chaos, more the touch-the-leftover-pizza-that-I-paid-for-with-my-budget -and-I’ll-stab-you-with-a-fricken-fork kind.

To make matters even worse, the agency’s budget changed with political winds, leaving the “less important” support functions (such as IT) with minimal resources. Massimo certainly didn’t expect to find top-of-the-line workstations, but he was astonished to learn what they considered top-of-the-line: a seven-year old Pentium II-350 with 64MB of RAM. Actually, compared with the rest of their technology, those were top-of-the-line.

The bulk of the agency’s network was at the central offices, though large portions were strewn across a hundred or so divisional and local offices. These offices were tied together through VPNs, leased lines, hard lines, and probably tin cans with a string. Internally, the inner-office networks utilized 10mbit hubs for communication.

And although Central IT managed each office’s servers, virtually every server was a different make, model, and configuration. What they almost all had in common were the basic specifications: Pentium II with 256 MB of RAM running Windows NT.

The Task At Hand

After learning about the Old, Bad, Ugly, and Unsupported setup, Massimo was glad that his job did not involve re-organizing the chaos. In fact, he was brought in help solve one specific problem: configure some servers on the network to relay outbound email messages to a central server instead of sending them directly through the internet.

Whoever had originally set-up the remote offices – presumably, the Lowest Bidder – decided to try a different approach for each office’s email. The central office used the infamous Exchange Server 5.5 (running on, impressively, only 256MB of RAM). Some remote offices had their own Exchange server set-up while others simply had their users download email from a POP3 server. Backup for the remote sites were virtually non-existent, leaving everyone’s mailbox (and important documents) a harddrive crash away.

Though email relaying would be a simple change to propagate across a Windows Server 2003 network, the agency had no plans to upgrade until at least Windows 2042 came out. Windows NT 4.0 on its own wouldn’t be too bad – just labor intensive. And if that were simply the case, the agency would have never brought in a consultant. Massimo’s job, as it turned out, was to find a way to configure about 100 of the NT servers that were in, as they put it, “Critical Condition.”

Server ICU

Most of the “critical condition” severs acted as a do-it-all box. They were domain controllers for the local office and application back-end hosts. Some had a few Oracle databases installed. But they all had “some version” of IIS running and a Corporate Antivirus package that, somehow, hooked into IIS.

Rebuilding the servers from scratch was simply not an option. They all stored valuable data and ran key applications, but no one seemed to remember anything beyond that. All the in-house admins knew was that the servers had been working for years, and it was just best to leave them be.

Massimo would just have to figure out how to configure the SMTP Services within IIS to relay messages to the central server. After analyzing each and every server, he determined that there were three different states that a “critical condition” server could be in.

1. Actually Working!
Due to excessive luck, a small handful of the servers actually had the SMTP Services installed; they would just needed to be left alone and could hopefully survive long enough to relay some messages before reaching the Big Domain Controller in the Skies

2. Exchange’d! A long time ago, one of those Lowest Bidder contractors installed Exchange 5.5 and, after quickly realizing that it slowed down the box considerably, decided to uninstall it. And by “uninstall”, I mean “manually disable any service that looked Exchange-related, delete whichever program files Exchange might have used, and changed a bunch of random registry entries without having any clue about what he was doing.” Blowing up the server would have probably done less damage than that.

Attempting to install SMTP Services on such a server would result in a “Exchange must removed before installing” message. Which, of course, would not be possible since it wasn’t actually there. Trying to put Exchange back on the server so that it could properly uninstalled would be equally futile. While Exchange lay in that sort of limbo, screaming in search of its revenge, getting the server relay any kind of e-mail was definitely impossible.

3. “Some Version” of IIS. You may have noticed the quotations around “some version”. This doesn’t refer to 2.0, 3.0, or 4.0 – but that other version of IIS called Personal Web Server. PWS is a stripped-down version of IIS that shipped with the NT Workstation Option Pack and a version that can’t be installed on NT Server. Except that it was installed. Somehow.

Massimo was never able to figure out exactly how it was installed, nor, more importantly, how to uninstall PWS without losing all of the configuration data like virtual directories, permissions, fie types, and so on. Or, for that matter, the tightly-integrated anti-virus software. He considered these servers to have a “chance” of mail relay.

We'll Get to It Soon

Fortunately for Massimo, after presenting his findings about the “critical condition” servers and the level of effort required to configure them to relay outbound email, the agency decided to push the change into the following year’s budget. When 2006 rolled around, they shelved it until 2007. And then 2008. They were so gonna do it in '09, but with the cutbacks and everything, it was pushed back a little longer. But 2010 is looking good!

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!