The marketing firm managed the web presence of several large banks and needed a Unix admin. Nick had spent the past decade running heavy HP-UX servers in the banking industry. It seemed like a very natural fit, and Nick thought he was going to enjoy the faster pace in a smaller firm.

The firm, as he learned, was broken into two major branches: consulting and everything else. Everything else existed to keep consulting happy, since consulting pulled down the big bucks. On Nick’s first day, his boss Ted introduced him to Larry, one of those consultants. Larry had some very important things to tell everyone who worked with the server team.

“There are two Windows boxes in the data-center,” Larry explained. “Those are mine. Do not touch them. Ever. Nod if you understand.”

Nick knew nothing about Windows and had no interest in supporting Windows servers. He nodded, but Larry continued. “You see, I’m the sort of person who was self-trained. My degree is actually in psychology, you see. I’ve had to learn everything in IT the hard way, and that gives me a much deeper and more practical understanding than you. You probably can’t even program in Visual Basic like I can, so just stay away from things you can’t understand.”

“Oh,” Larry said as he started to turn away. “One more thing- those boxes actually need a hardware upgrade. Could you be a sport and order me new servers for the rack? That’s your job now, isn’t it?”

It was Nick’s job. Since Ted told him to keep Consulting happy, and since Larry was Consulting, Nick did as he was told. He ordered Larry two new rackmounts loaded with more RAM than a goat festival and enough power in the CPUs to launch a goat festival into orbit. And for six months, that was the last Nick heard from Larry.

When Larry reappeared, he had a new request. “That hardware you ordered is already going bad. The servers keep hanging and crashing and grinding to a halt. I constantly need to reboot them. Order new ones.”

Nick gained a few important facts during those monhts. For example, the servers he had replaced for Larry were only three months old. And those servers replaced something only eight months old. And before that- well, it seemed there was a problem with Larry’s servers, but it wasn’t the servers.

Nick dug up an admin login for the Windows boxes, and armed with enough sense to call up the task manager, Nick started his investigation. He logged on to the servers that he was told never to touch, and then immediately felt a need to go wash his hands. Those two boxes were so loaded with viruses it was time to call in a nuclear strike to keep the infection from spreading. It was a smorgasbord of vile computerized diseases: trojans handed control of the system over to commands from certain .ru domains, a family of spambots churned out message after message. There was even an IRC server that was doing healthy trade in copyrighted material where the users mostly spoke Estonian.

Oh, and when the CPUs had time to spare, it also served up customer account information from the banking website the boxes were actually for.

Nick collected forensics and sent them to Ted and Larry. He suggested wiping the machine and reinstalling the OS from known good media and then patching the hell out of them before connecting them back to the network. He also mentioned that since this involved banking data, they’d have to file a disclosure for the breach.

Larry didn’t have much to say about this, other than, “No .”

Ted explained the situation: “We’re just going to run a cleaner tool against the box. Larry found something on Downloads.com that should get the worst of it. Since we aren’t taking the servers offline, we don’t need to file a disclosure.”

“Um… breach disclosures don’t work that way…”

“In any case, we can’t keep those boxes patched. The customer won’t pay for it. Consulting charges by the hour, and patching boxes is considered non-value-added. And Consulting doesn’t want anyone messing with their servers, so leave it alone.”

Nick went back to his desk and thought about what he had just seen. He decided that this really did need to get reported up the chain of command, and so went over his boss’s head. Things went quiet for a time after that, and then the memo came down.

Ted and Larry had been playing fast and loose with security and procedures for years. The hardware Larry was having constantly replaced for no real reason had been ending up on E-Bay after being decommissioned. Ted was the evil mastermind, Larry was simply incompetent. In a flash of institutional intelligence, both of them were fired, and Nick was given a promotion.

The downside was that Nick needed to learn Windows administration. At least he was confident that he knew more than his predecessor.

Image from Wikimedia, by Ward Moerman
[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!