Unfortunately for Elena, the laboratory information management system (LIMS) her team used was being sunsetted by the vendor, Initech. She's a biochemist working in a pathology department, and this software is vital to tracking the tests need, the tests already run, and providing critical notifications about abnormal test results.
Since Initech was sunsetting that product, the hospital system put out an RFQ for a replacements, and after a multi-year bidding process, offered the contract for replacing the software to Initech.
And thus, IniLab went away, and was replaced by IniHeal. Gone was the classic terminal interface that everyone had learned to use, and in its place was a "user friendly" GUI- a GUI that buried functionality behind fifteen clicks, had no meaningful keyboard shortcuts, and was constantly changing as they pushed updates, making it impossible to adjust to.
Also gone was the IniLab scripting language. IniLab's scripting language was how they had assembled workflows for laboratory processes. You could have fine grained control of which scripts tied to which kinds of tests, and even build custom workflows as one-offs, if necessary. They had many thousands of lines of code that needed to be ported over into IniHeal. The problem was IniHeal's approach to scripting was… well, bad.
First, IniHeal doesn't allow you write scripts. Instead, it has a pair of "rule" files. The marketing copy calls these "declarative", and promises that they're an upgrade. In practice, these are gigantic files that contain a set of rules that are evaluated for every test and every sample.
But the worse problem in IniHeal is how it lets you handle data. In IniLab, if you wanted the patient's age, you might reference the field by name: Patient.age
. In IniLab, you need to use the PPPPLL. What is the PPPPLL?
PPPPLL stands for "position" and "length". Instead of referencing a field by name, you have to reference the field by its byte offset and length. So Patient.age
might be [091001]
. And no, there's no easy way to create constants or variables to hold your PPPPLLs, you just have to know that [091001]
is age, and hope that the input data never changes its layout.
Now, one of the things Elena noticed while techs were on-site is that the length of the date time fields was 32 bits, and the data was seconds since the Unix epoch. So she asked one of the senior developers "Hey, is this software Year 2038 safe?"
The developer laughed. "I'll be retired by then."
The final insult to injury is that this software wasn't installed on any laboratory workstations. Instead, they had to use Remote Desktop to access it inside of a VM. While the environment was theoretically locked down so that the only program you could run was IniHeal, in practice it was trivially easy to get to a desktop inside the VM.
Curious, Elena poked around. She not only found the installation logs from her lab's deployment, but the installation logs from several other labs. These logs included various internal and private details, including account passwords. Somehow, log files from client installs had ended up in their installation media and were deployed to every client site.
Elena reported the potential data breach. The final finding was that the password in question was long defunct, which led management to decide "well, that's no problem then."
IniHeal has a slogan about equipping customers with innovative tools, but Elena proposes a new one:
"The Q in IniHeal is for Quality."