Russell had been a systems engineer for many years, but he'd only earned a nickname once while working for a comparison shopping site in West L.A.

The DevOps team had been struggling to automate configuration and deployment of servers when Russell arrived with his favorite tool: Puppet. It took some time to hammer the procedure into everyone's heads, but eventually nothing reached their production systems without being recorded in a Puppet manifest first, meaning servers could be built and re-built without manual intervention.

The DevOps team was now operating more efficiently than ever earning the flagship shopping site's developers kudos galore and Russell a reputation for getting things done. The missing link though was Network Operations. The company was happy to invest in engineering talent, but when it came to keeping the lights on and the backplanes communicating, Russell's boss's boss had a saying: "Our procedures are so unsophisticated that trained monkeys could run the NOC."

Russell had always thought that was a figure of speech until he met Craig and Carl. The two had been roommates in college and took a strange pride in telling Russell how they'd been too busy "hacking" (their word) to pass most of their classes. They knew how to use the UNIX command line, at least, so Russell was content to ignore them unless circumstance obliged him to go down to the NOC.

The dynamic duo had been at it for less than a month when a seemingly routine update push to five servers used by the development team failed. No problem, it should be a trivial matter of issuing a single command to Puppet, Russell thought, and it was. Well, for four of the problem servers anyway. The last one returned several errors complaining about its web server. "That's odd," Russell thought aloud, "the devs never mentioned a need to reconfigure Apache on any of these boxes..." Russell tried to log in to the problem machine, but the connection proved to be very unstable with his SSH connection dropping every few seconds, making recon impossible. Wondering what the heck could be wrong with the server, Russell tried again, this time as root. This did the trick and the session not only stayed alive, but it also handed back a massive clue right in the console:

THIS IS CRAIG AND CARL'S DEVLEPMENT[sic] SERVER

Russell wanted nothing more than to storm down to the NOC and hang both clowns out to dry, but he had a server to deploy. He found the problem with the Apache configuration and fixed it, but his inability to connect with his regular account bugged him. Scouring the server, Russell found a script called "scrubber.sh" that ran every few seconds, killing any SSH sessions that weren't owned by the users craig, carl, or root.

Through some miracle, Craig and Carl weren't fired - probably owing to the fact that their actions, no matter how devious they seemed, were the result of reckless inexperience rather than malice...and someone was probably in a forgiving mood. Now, you would think that Russell would protest this and push for their ejection from the company, but it was their attitude after the incident that was the most satisfying. You see, Russell had discovered Craig and Carl's mischief within an hour or two of them making the changes. The timing was a coincidence — but Craig and Carl didn't know that. They thought he just kept such a close eye on the servers using his magic "puppet tools" that they never stepped out of line again.

Sometimes, when Russell walked through the NOC, he could hear one of them mumble "puppet master" to the other, and he smiled.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!