Photo Credit: 'Jay Erickson' @ FlickrAfter a hard day of writing code at the Department of Justice, Sabbo settled into his favorite chair to watch the evening news. But instead of his usual glass of iced tea, that night it was a tumbler-full of Johnnie Walker Black on rocks.

Though it was a pretty normal day at work, the ride home was anything but normal. In the middle of his evening commute, Sabbo's boss rang his cell phone to let him know that "something big" would be on the evening news. Apparently, the Shame On You! "investigative" reporting team from the local news ambushed the Department director with question after question about Inmate #88172, Inmate #88172's family, and, most importantly, why Inmate #88172 wasn't getting the money the Department owed him. The director could say little more than "I don't know", which was exactly the sound bite Shame On You! wanted.

Sipping his Scotch, Sabbo watched the trademarked graphics fly across the screen with a crescendo of "shame, Shame, SHAME" yelled in the background. The report opened with a former prisoner - Inmate #88172 - being interviewed; he gave a heartfelt account of how he had spent the past fifteen years trying to build up enough money so that he could start a new life and that the opportunity to move on was being wrenched away from him because of government corruption that went all the way to the top. The news report accused everyone from the officers working at the prison to the new-car-driving warden to the mayor of being in on the conspiracy of skimming money from the prisoners' accounts.

Normally, Sabbo would have simply rolled his eyes, but this time he groaned, knowing that the next day would be very busy for him.

The next day, armed with the orders "use whatever resources are at your disposal before the no-good media airs a follow up story and condemns my career, but no pressure whatsoever", Sabbo dove straight into the current ERP system, pouring though five years of transactions made by the accuser. After four hours of mind numbing work, he learned that while in prison, Inmate #88172 purchased a chocolate bar like clockwork every Tuesday at 10Am. But aside from that, everything checked out: Deposits - Debits = Final Disbursement. Sadly, since there were no inconsistencies in the new system, it meant only one thing - Sabbo was going to have to do the unthinkable - he was going to have to ressurect the old ERP system and restore the prison's system to working order to be able to get to the bottom of things.

How the System Works

Since payment via a check is somewhat impractical behind bars, all earnings and contriutions from family and friends during an inmate's incarceration are directly deposited into a debit account. Using this account, prisoners can purchase items from the commissary. Whatever is leftover at the time of release is given back as cash.

In an upgrade from paper ledgers, the Deptartment of Justice's decreed that inmates' accounts were to be tracked using their chosen brand of ERP software that was to also be the muscle behind all "business"-critical processes in all of the local prisons. Being so ubiquitous throroughout the prison, it allowed the prison staff the ability to track and control inmates' cash flow from deposit, use in the prison, and finally, the cash disbursement.

Despite the stigma attached to most intra-government decrees, the system actually hummed along quite well for a number years. But after a while, its age was starting to show. The ERP software was five whole versions behind the most recent release, the underlying database seemed as if it were held together with duct tape and chicken wire, and, as if to make matters worse, each prison had their own servers.

So, rather than waiting to see whether the prison's ERP system would collapse or the IT Support staff would uprise and revolt first, the CTO decided to upgrade everyone to the newest version of the software and centralize at the same time. Sabbo was brought on with a team of top developers to build a system that would serve the needs of all institutions using one common instance.

The new central system went live and all was well. Just to be sure that nothing was missed, all of the old ERP systems were backed up and one piece of the old iron was kept around just in case it was needed. This was placed next to the new system with a tape-reel attached so that the backups could be uploaded and the data read in case of suspected problems. The old system was turned off and remained dormant, entombed under its dust-resistant shroud.

Opening the Tomb

Once the server was set up and dust was removed, Sabbo took a deep breath as he pressed the switch on the power supply. As if to warn potential interlopers of some terrible curse, the old ERP system's first act was to throw sparks out of the power supply unit. A quick call to the vendor and a new very-expensive power supply was overnighted. After installing the replacement, Sabbo was ready for round two.

The next step presented further difficulties but, this time, no vendor was needed. Blood, sweat, tears, and lots of cursing found the cause of the latest round of failures: one of the server's two CPUs was bad and memory bank #19 of 24 was faulty. It only took a full day to isolate the latter.

Next up, with the server finally booting to the OS, was to restore the database from one of the tape back-ups. And, wouldn't you know it, the server's tape drive failed with a pop and a wisp of smoke. Once another drive was located, fate decreed that since the tape hadn't been turned over in years, it would cling to itself on the spool, causing much of it to rip apart upon loading.

Finally, once the tape was meticulously un-clung and loaded, it took four full days of fiddling with the database before finally getting it to start up.

The End Result

The 300+ man-hours and cost proved worth it in the end: Sabbo could finally run a single report showing Inmate #882771's detailed transactions for twelve years, prior to the cut-over of the new ERP system.

And as it turned out, Inmate #882771 was not owed the $7.50 that he contends that he was due.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!