The ancient gods released their fury; wind and wrack raged outside. Wind and lightning smote the power to Hubert's office, plunging the offices and server room into darkness. Hubert was new to the company, and had never received any training on emergency procedures, but he did possess a degree of common sense. Instead of stumbling upstairs to the server room in the dark, he reached for one of the emergency flashlights and turned it on. Or tried to- the battery was dead.

Hubert stumbled to the server room in the dark, guided only by the blinking lights of WiFi routers scattered throughout the building, or the faint hints of light from the windows. When Hubert reached the electronically locked door protecting the server room, it was, well, locked. That battery was still working, although it was anyone's guess as to when it was last replaced. The voltaic cell managed to cough up enough power to grind the lock back with the speed of a stone door in the Temple of Doom.

The door swung open with an ominous creak. Within lay a viper-pit of cables, strung with no real sense of direction or purpose. Dust and cobwebs strung them together. The flickering HDD lights caused the cables to dance, like a writhing mass of centipedes on the floor. One by one, those flickering lights died. For whatever reason, the shut-down scripts that should have killed the servers cleanly hadn't executed. The UPS backups provided mere minutes of power. Helpless, Hubert watched everything crash.

After power was restored, it took days to bring things back online. Data had been lost, and the handful of backup tapes stuffed into a closet had a layer of dust that implied a century of disuse. While he waited for servers to recover, be restored, or rebuilt, Hubert spent a few minutes to file a requisition for a fresh set of UPS backups. He emailed it to the correct place, CCed the right names, and the patiently waited for it to be ignored.

A year later, the ancient gods had forgotten their fury. It was a lovely spring day; brisk, but not cold, with the sun shining, and the birds singing, and the UPS alarms screaming their bloody heads off. There was no power outage, and to the trained eye, nothing was overtly wrong. Aside from failing UPSes and servers crashing left and right. But if you ignored that, everything was fine. Once again, helpless and confused, Hubert watched servers cry out, and then crash as if they were suddenly denied power.

Hubert thrust his hands deep into the nest of cables, hoping to find a logical explanation for the problem. Instead, he found a delightfully illogical one. Each server had two power supplies- one for its primary load, and one for its UPS backup. There were multiple UPS backups, so only a handful of servers were ever drawing from one. The primary load, in a sane setup, would go to one of the many wall outlets. In this particular setup, they went to UPS #2.

UPS #1 provided backup power to half the servers. UPS #2 provided backup power to the other half. But it was also providing primary power to the same half of servers. And to compound the problem, UPS #1 was plugged into UPS #2. Poor little UPS #2, already well past retirement age, couldn't handle that kind of load. It tried, but the result was that it undersupplied the servers with unconditioned power, meaning the servers shut down automatically, fearing damage from bad power. With no spare energy to charge the battery with, UPS #2 had absolutely no backup power left; any outage would take down half their servers like a switch was flipped.

It took hours to dig through the cables to find this problem. It took mere minutes to juggle plugs around outlets to solve it. With the immediate outage solved, Hubert focused his attention on fixing the underlying cause. These UPSes were timeworn a year ago, and they were downright venerable now. He filed another requisition for UPSes, and this time, he CCed the CTO. He left that job well before they ever replied, but he assumes that they have their top men on it. Top men.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!