As an experienced IT professional, Drew had seen his fair share of network outages over the years.  He had protected his company’s network from storms, earthquakes, and even floods, but more often than not, the problem was from within.  His fellow coworkers found ways to bring down their computers far more frequently on their own, through a combination of sending virus-laden e-mails, unplugging cords, and in some cases just gross incompetence.

One afternoon, Drew was installing some software from the network onto an employee's computer when the system locked up for 10 seconds.

Thump.

Hoping this was just a fluke event, Drew continued the installation. Within minutes, the system had locked up again.

Thump.

At that point, Drew’s pager lit up like a Christmas tree with reports from multiple departments that the Internet was down.  Drew postponed the installation he had been working on and ran to the nearest working terminal.  He pulled up a network health check and was greeted with a large spike in the connectivity graph and huge amounts of packet loss.

Thump.

It was not often Drew talked to The Boss, the owner of the company, but today being a particularly unfortunate day, The Boss was already hovering at his desk, jumping up and down like a five-year-old with threats that 'heads would roll' if the problem was not fixed immediately.  The Boss proceeded to lecture Drew on how much better his nephew would do in the role of  IT technician.  Drew tried his best to tune out The Boss's rants and focus on the issue at hand.

Thump.

"We have run every test we could think of.  We do not know the cause but it is probably a virus spamming the system", a fellow IT technician, stated.  "Something like this happened a few years back and the best solution was to take down the network, clean the computers, and then turn it all back on".

Drew was not fond of catch-all solutions when a fine instrument would do, but he was short on time and The Boss was breathing down his network.  He gave the green-light to clean the computers and a few hours later, they were ready to turn the network back on.

Thump.

Within minutes the network was down again.  At this point management had left for the day, so Drew had some more time to consider the issue.  He regrouped with his team and decided to branch out from the main server room, bringing each group of computers back online one at a time.  They were about halfway done, with the network was still holding, when Drew came across an odd cable that he did not recall installing.  On a hunch, he plugged the cable in.

Thump.

"Ah ha", Drew thought, "We have found the culprit".  He asked one of his co-workers where the cable had come from.

"Oh, The Boss’s son asked us to set up a cable last summer," came the answer, and Drew winced.

The Boss’s son had been an intern, and done a "Spectacular job", as The Boss had put it.  Drew discovered the cable was connected to a computer in a forgotten room in the bowels of the company, where The Boss’s son had spent the majority of his internship playing online games.  Drew began his examination of the computer.  Despite it being a ‘complete mess’ of games, adware, and, yes, pornography, there was no virus, no Trojan, nothing that would explain the network outage.  He decided to test a theory, and plugged his laptop into the network through the reconnected the cable.

Thump.

"Ah ha," Drew thought. "It must be the cable".  He was nearing his goal. He traced the cable back to the switch, one section at a time.  It was near the roof that Drew located the object of his search: the cable ran straight into a shiny, new compartment.  He opened up the compartment and his eyes beheld a monstrous sight:  a large section of network cable had been looped and tied with electrical tape, adorned with wire nuts, and spliced into an electrical cable.

Drew called his team over.  "I think we have found the source of the problem," he announced.

A few weeks before the network crashed, some high winds and rain had damaged a section of the roof.  The Boss had called in his brother-in-law’s company to patch up the roof, his uncle’s company to repair the plumbing, and his brother’s company to repair the electricity.  The Boss's brother repaired the wires and, finding the phantom cable, "helped out IT" by "repairing" it.

The day of the outage, The Boss's brother had been reviewing the circuit breaks and noticed one of them turned off, so he flipped the switch without telling anyone and went to lunch.  The inactive breaker had been keeping his handiwork quiet for weeks, and by turning it back on the cable became active, sending a jolt of electricity down the cables, overloading the network.

"I don’t see what the big deal is", he later told Drew.  "It’s just another wire!"

Drew has long since left the company for greener pastures, eventually replaced by The Boss’s nephew.  But he still keeps the spliced section of cable in his desk drawer, in part because no one believes him when he tells the story, but also a reminder to 'always expect the unexpected.'  Every once in a while, if you listen very carefully near his desk, you can almost hear the faint sound of...

Thump.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!