Rex had just been hired on with a large retailer as a Puppet Automation Engineer, tasked with using Puppet Labs to automate deployments of some SAP-py, enterprisey software. He was paired up with another Puppet Automation Engineer, Alexi. Alexi was the expert, and he was in charge of automating the company’s Sarbanes-Oxley (SOX) auditing.

Alexei was a firm believer that the Customer Is Always Wrong, and Alexei Knows Best. As a consequence, he thought that any requirements he didn’t like could be changed to arbitrary ones he did like. If the customer wanted a report that provided some summarized sales numbers for the year and he thought that was stupid, he’d instead give them a report showing their top product’s Line-Of-Code count divided by the Dow Jones Industrial Average for the month. If they wanted to slice-and-dice their customer database by demographics, he would code up a line graph relating the number of characters in their last name to the average nightly lows on their date of birth.

SarbanesOxley
These are Sarbanes and Oxley, but you can imagine these are the characters of this story.

This caused huge problems with the SOX auditing. As he worked on the requirements-not generated by users, but dictated by US law- Alexei replaced “stupid” ones with his own “expert” ones. After a chain of angry emails, angrier phone calls, and downright apocalyptic meetings, management finally got Alexi to deal with the actual requirements. Exactly as written.

Exactly. As. Written.

From what Rex could see, much of ALexi’s implementation was deliberate malicious obedience. For example, many rules were given the same name, causing their auditing engine (which ran through the rules in alphabetic order) to give varying and nondeterministic results as the rules were run in a different order each time, often overwriting each others’ output.

What really amazed Rex was the incident that finally convinced his coworker to leave. Alexei was automating another piece of enterprisey software deployments- Initech’s “SalesManagerPoint”- with Puppet, using an existing (and broken, unsupported and officially deprecated) module designed specifically for their case. When the module didn’t work, Alexei doubled-down and tried to fix it, adding layer upon layer of expert complexity, meanwhile ignoring key requirements because “The module doesn’t support that!”

Frustrated, management brought in a Highly-Paid Consultant, named Harry. Harry was intimately familiar with Puppet Labs, and within a few days, proved the point by fixing many of their automation-pain-points. Everything was great, until Harry discovered Alexei’s work. Harry was also familiar with Initech’s SalesManagerPoint. “That’s a very tricky one. I can take a quick glance at it, but we normally will not touch that without a minimum six-month contract.”

As soon as the words were out of his mouth, Rex stopped paying attention to his work, and started paying attention to Harry and Alexi. Their disagreement escalated into a hilarious shouting match. Harry tried to tell Alexi he was using Puppet, the module, and SalesManagerPoint all the wrong way, to which Alexei would respond with screams of “But your so-called ‘right way’ is stupid! My way is better!”

Alexei was angry enough to quit on the spot and Rex inherited his work. Harry refused to go anywhere near anything Alexi had touched without renegotiating the contract. After reviewing Alexi’s work, Rex decided on ignoring it completely and built up a new system from scratch. Several months later Rex had a working Puppet automation that actually met all the customer’s requirements, without using Alexei’s work and without using the broken Puppet module. No expert required.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!