Shortly before the global banking system chucked itself out of a 30 story window, one large bank purchased another. It may be a common financial transaction, but from an IT standpoint, the problem of integrating these two systems is always unique, always cumbersome, and fraught with problems. Sajid was one of the lucky individuals tasked with the great undertaking of moving these banks to one information system.

Sajid’s business contact at the new acquisition was Karel. The latest round of requirements from Karel described a system to track the movement of trade data from his office, in the Netherlands, to Sajid’s, in London. The underlying business logic was simple, but the nature of financial data added layers of complexity. Moving data around was insufficient- there needed to be auditing, tracing, regulatory compliance. The project needed a risk management plan, the tool itself needed a clearly documented backup and recovery plan, and on top of all of that there was the simple management overhead of doing anything in a large organization.

Sajid’s finished estimate for the project contained a large number of digits. Karel blanched at the expense, thanked Sajid for his time, and scurried back to Rotterdam. Sajid assumed that he’d receive a budget in a few days.

Instead, everything was silent for months. The original deadline for the migration project slipped past, and a new project called “Tail Migration” began. When that deadline became a historical relic, the “Post-Migration” began, and mutated into the “Post-Tail-Abaft-Ongoing-Terminal Migration”. The ever-changing nature of integrating two systems nobody understood kept Sajid busy.

Then £14M vanished. Someplace over the North Sea, the data ceased to exist. Given the number of integration packages which handled the records, and given how many of them Sajid was responsible for, the missing money became Sajid’s very large problem.

Sajid started by searching the mainframes for the transaction data. He moved over to the MQ logs, the BizTalk audit trails, and the couch cushions. All he could find was a fourteen-million-pound-shaped-hole in their records. Suggesting that it might be a rounding error, or a missed decimal point didn’t make his management happy.

Sajid called Karel. “Oh, yes,” Karel said. “We did have an issue with the Bankieren Integratie Tool. I had to delete the code and rebuild it. Perhaps some transactions were lost?”

“The what? Wait- wasn’t that the estimate request I did? We never built that for you.”

“Well, you wanted too much money. We only need the system until migration is complete. We found an outside contractor who worked more cheaply. I’ll send you a link to the network share on my computer.”

Sajid’s mouse-hand twitched with terror and rage. He opened the network share and saw what he feared he would see- 8GB of data split across 6 Access databases. It was clearly “backed up”: a sub-folder on the same share called “Reservekopie ” contained older versions of all the files.

Sajid poked at the files like they were made of pure nitroglycerine and might explode if he sneeze. Despite being a mass of VBA, clumsy SQL queries, and being mostly written in Dutch, Sajid still recognized some of the functionality that he had estimated months prior. Like an old hard candy, forgotten under the couch, it had gathered layers of dust, dirt, and garbage. New features and reports had been added. None of it was audited. None of it was tracked.

There were nearly a billion pounds worth of transactions sitting in that database. Sajid had no idea if any lived in some other system, or if this were their only record.

Sajid copied the application and picked apart the database like a coroner examining a murder victim. Most of the data resided in split files- Access capped at 2GB of data per file. All of the code resided in one file called “DeCode.accdb”. In addition to holding code, it held scratch tables for some of its work.

It was impossible to be certain, but Sajid’s best guess was that some bug in the underlying VBA code threw errors, and convinced Karel that the database was “corrupt”. Karel deleted “DeCode” and replaced it with his “backup”. The £14M Sajid needed to find caused the error, and had been deleted along with the scratch table.

The data was gone, and with it, the money. There was no IT solution here, and Sajid needed to turn this over to accounting. They needed to perform an audit to rebuild the transaction data. Banking regulators would need to get involved. Honestly afraid of the coming fallout, Sajid went to his boss to deliver the bad news.

Little did he know that his boss had bigger problems- the person responsible for running their close-of-business batch jobs had accidentially deleted the entire process queue. Their bank was effectively closed until the very finicky mainframe scheduling system could be reconfigured. Suddenly, losing £14M wasn’t much of a problem.

Image via Wikimedia Commons. Source.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!