Datacenter Cloudwatt

The bank Edward worked for had a datacenter problem. Said datacenter resided in the basement of their headquarters. Over a twenty-year period, it had been expanded twice, and now covered the entire floor. There was simply no place left to go. The datacenter contained everything from state-of-the-art racks to $10 Ethernet hubs that no one had touched in a decade, and many of these mission-critical components were situated directly upon the floor. Every other week or so, some technician would trip on a cable and knock out a server or switch.

While the bank had plans for a new headquarters and datacenter, construction had been postponed twice. The most optimistic estimate was that work would not begin for another 2 years. In the meantime, Edward and his IT team were tasked with figuring out how to squeeze more space out of the existing datacenter.

The team's first move was to set up virtual Windows servers, so they could throw away some of the older computers cluttering the floor. Spreadsheets were compiled to track server names, business group owners, locations (ex. "DC room #2, third gray desktop on the floor from the left"), IPs, applications, and when virtualization would occur. Unsurprisingly, there were hiccups in tracking down owners, figuring out what applications ran on which machines, and whether those applications would work properly in a virtual environment. Overall, though, the virtualization push was successful, allowing much precious basement real estate to be reclaimed.

But, wouldn't you know it? There was a problem with "DC room #2, third gray desktop on the floor from the left." It'd been successfully virtualized, and the application running in the virtual environment had not yet exhibited any problems. Just in case, the old server remained temporarily in the datacenter, with a "DECOMMISSIONED" sign taped to it. Everything seemed fine—until Edward received an IP collision alert on the beginning of the next month. Someone had fired up the old server, which began butting heads with its virtualized clone.

It was easy enough to shut down "DC room #2, third gray desktop on the floor from the left." Less easy was figuring out who had turned it on. Over 100 people had unrestricted access to the datacenter. Server logs were of no help, either; whoever had logged in had used one of the general maintenance accounts that half the IT department knew the credentials for. Edward emailed everyone remotely related to that server, asking who might have turned it back on. To his complete lack of surprise, he received no reply.

The beginning of the next month, there was another IP collision alert, along with several angry phone calls from users. Edward went down to the datacenter and found the server back on again. Someone had carefully un-taped the "DECOMMISSIONED" sign and moved it a few inches over to access the server's power switch.

Edward appealed to his manager: there was no trouble with the virtual environment. Surely it was time to get rid of "DC room #2, third gray desktop on the floor from the left?" Alas, no. His manager explained that, due to how many departments relied on this server for reporting, they couldn't afford to get rid of it if there were ever an issue with the virtual environment. But he did allow Edward to do the next best thing: unplug the server and hide it behind a cardboard box full of power cables.

The beginning of the next month, there was no IP collision. However, there was a frantic call from one of the 24/7 datacenter monitoring personnel, claiming that one of the servers had been stolen. When Edward and his manager inquired further, the details came out. This person had mapped a drive for one of his monthly reports to "DC room #2, third gray desktop on the floor from the left." Edward's cloning script had somehow missed this share. Whenever this person had to access his report, he would receive an error. Since datacenter hardware issues were common, he would think something had happened to the server, and would simply fire it back up. Once Edward had hidden the server, he'd reached the only logical conclusion.

"You saw the 'DECOMMISSIONED' sign," Edward pointed out. "Didn't that tell you anything?"

"Half the servers in the datacenter have had stuff like that on them for years!" he dismissed.

"What about all the emails we've sent out?" Edward asked.

"I don't read emails with too many recipients," he replied. "If anything really important happens, I'll get a call about it sooner or later anyway."

"Why didn't you ever file a support ticket?"

He wrinkled his nose. "You never know who'll pick it up! Some of the people who work here are idiots."

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!