photo credit: Strupey @ flickrAlex’s phone interrupted dinner with his Fiancée. It was the office. Again. The phone server had gone down, and that meant callers were being greeted with a busy signal instead of the friendly auto-attendant. As one of the few employees capable of toggling a server power switch, Alex asked for a doggy bag and headed to the office to reboot the server.

It was the third weekend in a row that the phone server had gone down, and Alex was getting a little tired of the mid-weekend interruptions. He didn’t mind providing the occasional off-hour support, but this was getting a little out of hand.

“Don’t worry,” his boss assured him on Monday, “I’ve figured out the problem. It’s heat ventilation, and I’m taking care of it today!”

His boss’s enthusiasm was as reassuring as an apology from the cable company. Though his boss held the CTO job title, his zero-years of previous IT experience shined almost every day. But his ineptitude never seemed to bother his boss’s boss, though that might have had something to do with the fact that they were brothers-in-law.

“Are you sure that’s what it is?” Alex responded, “the other servers didn’t go down, and it seems like there’s a lot of airflow in the server closet.”

“Go touch the phone server,” he challenged, “you could fry an egg on the thing! I’m going to add some airflow.”

And that he did. Before lunch, the server closet door was propped open and a noisy little circular fan was aimed directly at the phone server. Alex seriously doubted that it was a ventilation problem, but he’d since learned that the best course of action was to simply let the CTO learn this himself. That, and everyone enjoys a little schadenfreude now and then.

A few weeks went by and, to Alex’s surprise, the problem had finally been solved: he had not received a single evening or weekend “phone server reboot” call. Of course, the server still was overheating, but it wasn’t Alex’s problem: the CTO showed the receptionist how to reboot the server, and the receptionist had been coming in to toggle the power switch.

A Real Solution

The “receptionist fix” was apparently not the long term solution. One day, Alex spotted two workmen installing an air conditioner in the server closet. They had a ladder, a toolbox, some power tools, and a gas can. He did a double-take. A gas can? For an air conditioner?

As it so happened, building management had turned down the request for additional cooling in that room, so the CTO decided to violate the lease and install one anyway. After all, it was only a $2800 in hardware and about $700 or so to install. And that’s where the gas can came in.

You see, in addition to pumping out a crisp and wonderful coolness, air conditioners remove a large portion of water from the air, and that water needs to go somewhere. Usually, the condensation is simply collected and pumped through a tube to a drainpipe. No mess, no fuss.

But there was no plumbing in the server closet. Or near the server closet. The closest drain was a solid forty yards, and building management certainly wasn’t going to run extra pipe for an air conditioner they didn’t know about. The five gallon gas was the perfect workaround, so the installers claimed, and at worst it’d have to be emptied once a month or so.

It turned out that the “once a month” estimate was a little off, though to the installers’ credit, it was the first time they had used a gas can to collect condensation. Emptying the can was more a weekly duty, and one which fell to Alex more often than not. Five gallons of water was a bit much for the poor receptionist to carry... especially since she was still on reboot duty.

The air conditioner offered no real help to the crashing phone server, but the CTO half-expected that. In fact, he had already devised a plan: Alex was told to turn down the thermostat five degrees a week until the unit could overpower the heat that was being pumped into the room on the weekends. Consequently, emptying the can had become a daily task: two days of condensation would nearly overflow the gas can, and any more would certainly flood the server room.

For the week days that worked just fine, but it created a bit of a problem on the weekend. Since it was Alex’s job to empty the can, he devised a solution to avoid coming in on Sundays. Not server related of course, as that clearly fell in the CTO’s territory and the CTO had already come up with a solution to that problem. The solution was simple but effective: a bigger can.

The CTO ordered the largest cooler he could find. It was a hundred-quart monster that would hold five times as much water as the gas can. And it had wheels so that it could be rolled with ease to the break room, and then emptied.

Within a week, or “three cans” as Alex was starting to compute time, the cooler finally arrived. There was no room on the floor of the server room, so he brought in his trusty drill and popped a hole through wall into the adjacent coat closet. The massive cooler was positioned there with a hose leading through the wall from the server room carrying the water. It was the perfect workaround.

Empting the Cooler

Although the condensation collection capacity had increased fivefold, so did the weight. And twenty five gallons of water is a good two hundred pounds or so of weight.

It took a while, but Alex figured out to how to leverage his body weight to prop the cooler on its wheels, turn it around, and push it to the nearest floor drain. And then he emptied the cooler the next week. And the week after that. And the week after that. It was tiring, but it was some good exercise and at least he wasn’t on server reboot duty.

All the while, the phone server kept crashing. The CTO was convinced it was still a heat ventilation, but the air conditioner was already at its lowest setting of fifty degrees Fahrenheit. So he brought back the small circular fan. He bought a bigger fan. He brought an even bigger fan. He even tried keeping the doors open, which had the unfortunate consequence of keeping the area around it (including the poor receptionist’s desk) really cold.

Nothing seemed to solve the problem, and the CTO finally capitulated. He tasked Alex with fixing the problem. Not the condensation problem. Not the cooling problem. But the actual crashing server problem.

Alex was not much of a hardware guy, but he had built a computer or two in his day. The phone server was basically just like any other computer, just with a large bank of modems and phone cards. Not knowing where else to start, he pulled the side off of the large tower case.

He was greeted with a small plume of dust, and then a motherboard, more dust, a video card, dust, two hard drives, dust, a floppy drive coated in dust, power supply humming away, and CPU covered in dust. He did a double take. A CPU covered in dust!?

Looking closer, Alex saw that a wire from the power supply was tangled in the CPU fan. He stared dumbfounded, and wondered if he should spend the next day or two at the bar “fixing” the problem. Wanting to get back to some actual work, he instead untangled the wire, gave the fan a little spin, and watched it spin up like new.

After putting the case together and dusting off, Alex took a good, hard look at the cooling kludge that he had partially created. A large, A/C unit on the ceiling blasting cold air with a tube running through the wall to a large, Coleman cooler that was propped open with a binder clip – the same binder clip he used to crimp the hose while he emptied the cooler – and of course, a large fan circulating even more air the small server rom. He wondered if he should be proud or ashamed of this kludge.

Opting for a sort of embarrassed pride, Alex turned off the A/C unit, moved the fan out of the server closet, closed the doors, and went back to his desk. The server hasn’t crashed since.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!