Photo credit: 'Csaba_Bajko' at Flickr "Alright! 11:02 a.m...Another five bucks for me!"

Once again, Ricky Fine had hit it big in the network technician over/under betting pool for predicting the time of the daily network outage. It occurred around 11am, every day, but not precisely. The network would pulse on and off around 60 cycles per minute, but again, not exactly.

"Any results from the packet sniffer?" asked another technician.

"Nothing, not even so much as a stutter," replied Ricky, "You'd think for as much money as this thing costs, we'd get a hint as to what's going on."

The technician sighed, "Well, we'll just try again tomorrow. What do you say? Next one's going to be before 11 a.m.?"

"Definitely. We're due for an 'under' this time around," agreed Ricky.

Days passed and with their tools failing them, Ricky and his fellow technicians finally had to resort to brute strength analysis. This meant disconnecting different network segments every day and was by no means an easy task.

The Hardware

The thing about the hardware that Ricky and his co-workers supported was that it came from an era before CAT5 and fiber optics. Their ethernet was a swarm of coax cabling and BNC connectors, usually run haphazardly through buildings that were never designed for it. Also, as if to add insult to injury, the company exclusively used hubs for their internal network.

There’s a good reason why hubs have since been relegated as relics: unlike switches, which intelligently route packets to the appropriate destination, hubs simply repeat all packets from all connected devices to all of the other connected devices. This, along with wiring factors made the outage develop into a riddle, wrapped up in an enigma, encased within a rats’ nest of cabling.

Brute Force Pays Off

Finally, one day, they located the problem segment as the network was going down - the fifth floor. They checked every bit of cable on the line, working through the night testing impendence, frays, an even going so far as to swap out every network card on all of the PCs on the floor. Having done everything in their power, Ricky and his crew felt confident that the problem was solved.

No such luck.

The next day, 11 a.m. rolled around and a couple of minutes later, there was the regular outage. Like a pack of bloodhounds, the technicians fanned out to every cable run connected to the floor's hub.

Ricky was under a cubicle desk looking for the cable run when he heard, "Thump, thump, thump..." coming from the other side of the cubicle wall. It was about every second, but not exactly.

When he emerged and peeked around the corner, he was met with the department's resident coffee expert. He could extol the virtues or curses of every kind of coffee made and every roaster in the area. And as it happened, he had just returned from percolating his morning dose.

Carefully observing the coffee connoisseur, Ricky watched him take a large sip of coffee, set his foot on the BNC connector, and nervously roll it back and forth a few times. A few second later, he'd take another sip and tap the connector again.

It didn't take too long for Ricky to realize that, every day, he'd get hopped up on java, direct his nervousness at his work and absently roll the BNC connector like others would drum their fingers. The outage would provide enough of a break that he'd calm down a bit and leave that innocent connector alone for the rest of the day.

Ricky went ahead and moved the BNC connector out of the line of fire and the next day, no problems.

The Aftermath

After the smoke cleared and business got back to normal, the final cost of the outage was calculated, and the executives were furious. From the rented network sniffer to the hours of productivity and data lost during the outage, someone was going to have to answer how they could be so naive as to not be able to find that the problem was caused by the nervous gyrations of the local caffeine junkie.

Ricky's group did their best to emphasize that their infrastructure was at fault rather than the competence of the technicians. The execs scoffed at their explanation but, when the cost of new cable runs and upgraded network equipment was compared to the cost of the outage, they relented and agreed to upgrade the infrastructure. After all, there were a lot of coffee drinkers in the company.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!