Sometimes in IT, we have to be dicks. Like private dicks, I mean. Like Dick Tracy dicks. What did you think I meant?

Well, DickNick H. found himself wrapped up in an IT investigation that not only had intrigue and mystery, but trickery and knavery! It all started when Nick was working on a web project at his firm in the UK. The setup phase had gone well; they had a dev server, source control, and backup servers. They'd hired contractors to make some modifications to the building, adding network ports, cabling, and installing server cabinets. They received a 1Tb NAS drive, installed it, set it up to receive backups, and all their testing of the backups went well. Nick insisted on a trinagulated backup plan, so all their files would be backed up remotely, as well.

One day they needed to restore some files from the backup, and they noticed that the local backup failed at 1:30 AM, while the remote backup had completed successfully. The next day it ran fine. And the day after. Then it failed again due to the NAS drive powering down at 6:35 PM.

Nick began his investigation. He changed the script to run at 6:30 PM and stayed late after work to see what would happen. He and his team watched the backup run perfectly, so they all went home happy.

The next three days the NAS drive would power down, the local backup would fail every time, but when they'd come in the next morning, the drive would be running with no indication that anything was wrong. Nick knew something was up, so he stayed extra late, waiting until 11:00 PM for the backup to run on its normal schedule. It ran perfectly, so Nick went home at 1:00 AM happy (but tired).

When he groggily came in to work the next morning, the log files showed that the NAS drive had been powered down between 1:00 AM and 9:00 AM. Nick checked every lead, plug, socket, and cabinet temperature he could, and everything seemed OK. He wrote a polling script that would hit the NAS every two seconds that would play SIREN.WAV when it failed. I would've gone with CANYON.MID personally, but I guess they do things differently in the UK.

At 6:18 PM that day the siren rang out and everyone froze. Nick ran to the NAS drive, and began asking people that were near the server room what they were doing. "I'm just signing out of Windows," one said. "I just put my coat on," another said. "I just turned the kitchen light off."

Unbeknownst to Nick, the electrical contractors had grabbed the wrong wire for the server room (which was right next to the kitchen), using the same circuit as that which ran to the kitchen light. When people would leave for the day, they'd turn the light off and the drive would power down. When Nick would stay late to monitor the backup, he'd leave the kitchen light on until he left.

After having the electrical contractor come back and fix the problem, Nick went home happy.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!