The chemistry department is always one of the coolest places on any college campus. It's a place where students show up to class, and rather than getting boring instruction or lectures, their mad scientist professors mix chemicals together and cackle over a boiling beaker of awesome radness in front of the class. The rooms are adorned with exposed metal prongs with arcing electricity, torches, and fog machines. I should note that I didn't take chemistry in college, but this is what I'm pretty sure chemistry classrooms look like.

About 10 years ago, Bowie P. managed to score a job at his university's Department of Chemistry. It was awesome, too. He was given keys to the building, 24 hour access to all the francium he could dream of, and his job had him working between two buildings full of people he actually liked. Plus, just from reading the labels on doors in the buildings, he learned fun words like "photoelectronspectroscopy," "postneuroelectrondefenestration," and "floccinaucinihiliblackmetalipilificationbicarbonateblutausnord."

In Bowie's experience as a Unix admin, you really have to hit the ground running. When you're taking over a system, you have to consider that the previous administrator left to pursue full-time work protesting Microsoft launch events, went insane, died, or went luddite, leaving a mess for the next unfortunate administrator to clean up. Whether it's negligence, incompetence, or malevolence, the previous administrator tends to make things hard for his replacement.

In Bowie's case, though, he was fortunate enough to be shown the ropes by the guy he was replacing, Dave. Dave had made a full time job out of putting out fires that he'd started. He had an "if it ain't broke, don't fix it" type of mentality. As such, a lot of the system was half broken.

Bowie could tell that he'd be busy after just a few hours of sitting in with Dave. Very little had been automated; Dave took it upon himself to perform repetitive tasks that could easily be scripted. Fortunately, he'd automated one of the most common tasks: adding users.

User information was stored in an NIS/YP database that governed the whole department's login and ACL-ish setup. No one gets anywhere if NIS doesn't allow it.

Proudly, Dave demonstrated the Perl script used to add users. It would concatenate a username to the end of an NIS group definition and dump it to a text file. The text file would be pushed out through the network, where it'd ultimately be imported into the database.

Bowie saw a red flag. "Dave, what happens when the text file gets too long? I don't know that just tacking the username on to the end of the string is a great idea..."

"Eh," Dave shrugged, "don't worry about it." Dave tried to sound reassuring, but strangely Bowie just felt more concerned.

Due to his lack of familiarity with Perl and NIS/YP, Bowie didn't feel like he was in a position to criticize Dave's code. He continued to listen to Dave's instruction.

A few days later, Dave proudly showed off the backup scheme for the four-node cluster they had. Each system had a dedicated DDS2-DAT drive. "You know," beamed Dave, "there were no backups in place before I started." They didn't have a backup scheme before? Cool, thought Bowie, expectations are low. Plus there were backups; he had a safety net. Looking at the tape backup system, Bowie asked if Dave ever checked the tapes to make sure they were still good.

"Eh," Dave shrugged, "don't worry about it."

This guy's kind of a dick, thought Bowie. He reasoned he should just deal with it. This was his first time seeing a DAT drive in the flesh, so it wasn't his place to be critical of Dave's strategy.

When the time came that Dave was gone and the system was Bowie's responsibility, Bowie continued to learn the system. After a few months he was adding users with ease and had all but forgotten about his initial concern of truncated data. That is, until he tried to add a user that pushed the text file length over the limit for the database column. The database import failed, spewing out dozens of error codes that Bowie now knows better than his birthday, social security number, and middle name.

Within two minutes, his phone was ringing mockingly. *RINGRINGYOUBROKEITRINGRING* it cried. *RINGRINGTHEY'REGOINGTOFIREYOURINGRING*

Bowie hated computers. He hated himself. He utterly detested mother*RING*ing Dave. Wallowing in self-pity, he couldn't bring himself to pick the receiver up. No users could get into any of the systems until he fixed it — not even him. And he didn't have any answers. No ETA, no strategy to fix the problem, no idea even where to start. He tried to work without paying attention to the horrible, stomach-turning ringing.

Bowie frantically tried to debug a language he wasn't terribly familiar with on an 80x25 green screen. Finally, he isolated the line that failed. Sure enough, the list of user names was too long for the 1024 characters allotted in the database. But rather than clean it up or fail gracefully, the database stayed half-built. To make matters worse, Dave's script disregarded any return codes from the database, pushing out a broken compiled file through the network. Brilliant.

Not only was the database broken; it had been pushed across two buildings, the cluster, and 40 or more workstations. Bowie fought thoughts of professors and students preparing torches and pitchforks while he scrambled to get a fix out.

Bowie unplugged his phone, closed his door, and tried to restore from a backup tape. Dave had used the same backup tape over and over, to the point that the tape couldn't hold any more readable data. Totally useless. Dave's smug "don't worry about it" echoed in Bowie's head. Bowie started hacking. He did anything he could to get the system back online. "Hack, hack, hack, breathe, hack, blink, hack, hack, breathe, breathe, hack," he thought, furiously typing.

And it worked. After two and a half hours of hacking Perl, DB2, and NIS/YP, the system was back up. The group definitions were now broken into separate lines and the database was re-pushed across the network. Over the next week, Bowie learned enough Perl to totally replace Dave's jalopy of a script.

And still, as much as he would've loved to point the finger of blame on Dave, professional integrity prevented him from doing so. All of the blame fell squarely on Bowie's shoulders, and while he had some hurt feelings, it didn't hurt his career on the whole.

It has left Bowie a post-traumatic irrational fear of databases, though.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!