A few years back, Brian T took a position in the “Installations and Upgrades” department at a certain enterprise software company. Being that they were an enterprise software provider, installation and upgrades of their software could only be performed by highly-paid technicians. It was Brian’s job to support the technicians and provide them with scripts to help them do their job.

To get accustomed to the tools he’d be working with, Brian opened up the primary upgrade script. He was surprised to see the following at the top…

#/usr/bin/perl
$ActivateJoshsWorkarounds = 0;

As we’ve all seen, such variable names are rarely a good sign. Brian asked around to try to figure out who this Josh fellow was and why he had a variable.

Josh, as it turned out, was Brian’s predecessor and left the company out of “frustration with Development.” As one of the developers put it, “Josh just kept mucking around with our Database and we simply removed his CVS access.”

Figuring that he’d have a lot of Josh-undoing to do, Brian began the search for $ ActivateJoshsWorkarounds. The first block of code that utilized it was as follows…

if ($ActivateJoshsWorkarounds) {
 print "Call Josh at 440-555-0221 and tell him what the following output says...\n";

  &FindAndFixBrokenDBStuff;
  exit 0;
}

Brian marked that block for deletion and moved on to the FindAndFixBrokenDBStuff function:

sub FindAndFixBrokenDBStuff {
print "This code was removed because we have no clue what, if anything it does. We have also removed Josh's CVS rights until he can explain why he feels our databases contain broken stuff that needs to be fixed! Honestly do we tell him his stuff is broken?\n";
}

Brian quickly peeked into the source control history and saw that Josh’s code was replaced by one of the Developers. Eager to see what WTF’s he might find in Josh’s original function, he viewed a previous version…

sub FindAndFixBrokenDBStuff {

# Occasionally, there will be duplicate Primary Keys because Development
# chose not to enforce that constraint. We need to assign new keys, 
# otherwise the 4.02.1 Upgrade will seg fault!
... snip ...

# Clean out Orphaned rows, as the reports drill-down will crash with it
... snip ...

# Ensure that all "Received Dates" are actually valid Dates, otherwise
# the software will null this column for all rows!
... snip ...

# In some cases, the software will drop the ACL tables if there are 
# negative values. We need to clean it first so they don't lose data.
... snip ...

... snip ...
... snip ...
... snip ...

}

As he perused the code and tried to wrap his head around all that Josh’s script fixed, Brian received his first support call. One of the technicians encountered a rather bizarre error while doing a 4.02.1 upgrade...

ERROR: Primary Key Violation
Segmentation fault.
Segmentation fault.
Segmentation fault.
Segmentation fault.
Segmentation fault.
Segmentation fault.
Segmentation fault.
Segmentation fault.
Segmentation fault.
Segmentation fault.
ERROR: Segmentation fault.
Segmentation fault.
Segmentation fault.
Segmentation fault.
ERROR: UNEXPECTED END.

Fortunately, Brian had a pretty good idea where to start fixing it. He could tell, though, it was going to be a long tenure in “Installations and Upgrades”...