Comment On MUMPS Madness

This year’s Corporate Technology Expo was no different than the ones for years previous. Various departments gathered in the company’s large, wood-paneled group meeting hall and highlighted their top projects and initiatives that were completed during the past year. There was everything from the ASP-to-ASP.NET upgrade of the customer portal to the enterprise-wide implementation of COGNOS 7. The scene was a three-hour, seemingly unending procession of PowerPoint slides with enough laser pointers to take down an incoming ICBM.

Nobody would probably show, let alone stay awake, if it weren’t for the free coffee and bagels.

However, you knew that the light at the end of the tunnel was approaching when the AppDev’s “big guns” arrived: a cabal of four MUMPS developers headed up by a hairy yet balding man named Tyson. They were the brilliant minds behind the company’s core application, codenamed CASTLE. This year, they brought t-shirts for the audience.

Tyson went into the usual spiel about the impressive accomplishments of their application, the 99.9% uptime, and the tens-of-gigabytes doled out by the CASTLE system which fed data to the previous presenters’ systems. To wrap things up, Tyson revealed his “one more thing” – a brand new, home brewed combination web and mail server written entirely in MUMPS without any use of any common API. Basically, even with the grandstanding, the presentation was a friendly reminder that everybody’s livelihood depended on Tyson’s group.

Quick MUMPS Review

For those who aren’t familiar with exactly what MUMPS is, you’d be missing out if you didn’t check out A Case of the MUMPS. To summarize, if you’re associating MUMPS with the awful disease of the same name, you’re very close.

Quite possibly created as an exercise in sadomasochism, the “Massachusetts General Hospital Utility Multiprocessing System” was developed back in the mid-1960’s by Dr. Octo Barnett and with the help of federal grant money. Originally limited to medical information systems, MUMPS has found its way into other fields. Most notably, the financial sector.

To give a little more “real world” example of the horror that is MUMPS, start by taking one part [[International Obfuscated C Code Contest]], a dash of Perl, two heaping measures of FORTRAN and SNOBOL, and the independent and uncoordinated contributions of dozens of medical researchers, and there you go.

The Project

Everyone in Steve X’s department lived in fear of the CASTLE. While Steve’s background and job title may not shout “MUMPS Developer”, part of his job is to work with the quartet of diehard MUMPS developers who designed the system.

Thanks to the ignorance of management (and perhaps a “wink and a nod” here and there), Tyson and his “boys” had built CASTLE as a 350,000-line strong fortress of MUMPS over a period of more than twenty years. Tyson and his group had pigeonholing down to a science; everyone had relied on their system so much that they were untouchable and their group was impenetrable. Except for now.

“It’s taken a while,” the auditor said to Steve “but we’ve finally convinced Tyson to let someone go in and clean up some of the ‘junk’ tables that have built up over the years.”

“How’d you do that,” Steve smirked, “a note from God?”

“No, no, no. I pulled the Sarbanes-Oxley card on him. You’ll see a meeting invitation with his group for Monday morning. Good luck!” the auditor said as he dashed into the nearest hallway.

The following meeting with the quartet of MUMPS-men went relatively smoothly – probably from Tyson’s desire to get the auditor and Steve out of his hair and out of his system as soon as possible.

Steve got to work. His investigation could be compared to peeling back layers of wallpaper and seeing the horror it was really covering up. After a month of work had passed, he showed that, out of the 393 known tables, 225 were no longer in use. To make things even more interesting, many had been out of use since 1985 or earlier.

Included in the interesting finds were:

• A table called “GIVE THIS A GO” with zero records.

• A table called “Clinician Orders”, which was apparently used to record what drugs/medical procedures the clinician ordered for the patient at a given time. It had 133 columns:

Date
Visit
Clncn
Student
Trtmnt
Obsrvtns
CrtdUser
CrtDtTm
LstEdtUs
LsEdDtTm
[ un-named column ]
DscUsrSg
DscDtTm
DscSgnUs - Discontinued Signed-In User
AthCntUs - Authorize Controlled User
AthCntDh
N1amOrdr - 1am Order
N2amOrder - 2am Order
...
N11pOrder - 11pm Order
N12mOrder – 12 Midnight Order
N1amSgnt - 1am Signature
...
N12mSgnt  - 12m Signature
N1mDtTm - 1am Date/Time
...
N9mDtTm - 9am Date/Time
[ un-named column ]
[ un-named column ]
[ un-named column ]
N1pDtTm - 1pm Date/Time
...
N12DtTm – 12 Midnight Date/Time
N1amUser - 1am User
...
N12mUser – 12 Midnight User
N1amCmmn - 1am Comment
...
N12mCmmn – 12 Midnight Comment

That’s right: At least 5 fields for every hour of the day. God only knows what would be done to this poor table if clinicians decided they wanted half-hour (or worse yet, 15-minute) granularity.

• The calculation to determine drug pricing was extremely complex and un-commented. There were three different price fields:

PrcUntCs - Purchase Unit Cost
SpcRxPrc - Special Rx Price
SpclCost - Special Cost

Depending on a complex calculation (including such WTFs as “Is there an ‘@’ symbol in the field specifying the type of drug?”),any of the three price fields could be used and a variable markup added. For bonus-bonus fun, here is the actual code for the calculation:

 set cost=$s(spc:spc,1:puc)*1.3 ;cost + 30%
 set amt=qty/each*cost

 ... if drug falls into another category ...
 set amt=amt-(amt*.3)

Tyson claimed they intended to “take away that 30% markup,” not realizing that (X * 1.3 * 0.7) != X. He never quite explained the reasoning behind "amt=amt-(amt*.3)" instead of "amt=amt*.7".

• Compounding information on the drug table.

Since drugs can be compounded (i.e. made up of several other drugs), the programmer thoughtfully added these columns to the drug table:

  Item1
  Itm1Qty
  Itm1Lbl
  Itm1Prc
  Item2
  ...
  Item10
  ItmTenQt
  ItmTenLb
  ItmTenPr
  ItmLotNo - Item1 Lot#
  Itm2Lot - Item2 Lot
  ...
  Itm9Lot - Item9 Lot 
  Itm1Lot - Item10 Lot
  Itm1Expd - Item1 Expdate
  ...
  Itm9Expd - Item9 Expdate
  ItmTenExpdate - Item Ten Expdate

Fortunately, someone explained how related tables worked to this programmer before CASTLE needed to handle drugs made up of 11 items. Thankfully, the above compounding system was only used for 15 years or so.

The VISA Problem

After Steve’s database debridement project was completed, he was given one more task by the auditing group: search for and remove any credit card numbers that may have been stored in the system.

As it turned out, CASTLE had been storing customers’ full sixteen-digit credit card number for years, just in case someone needed to look up a transaction and the last four digits weren’t good enough. Eventually, Visa came out with their PCI requirements, making this kind of thing a no-no. So, to avoid a potential disaster and a hefty $500k fine, Steve built a scrubber tool to seek out and destroy the credit card numbers in the database, including the ones entered into the “notes” section of the client records.

Later, just before the scrubber was to be run in Production, Steve was shocked to find the following enhancement made to his code:

set ^tnc("scramble","notes",d0,d1)=data

In human-speak, this code is a loop where d0 is the clientId and d1 is the number of the note attached to the client’s record. data was the note containing a credit card number. tnc was someone’s initials. More specifically, they were Tyson’s initials.

What Tyson had accomplished was that he had inserted a line into the scrubber that effectively saved all credit card numbers to his personal database right before they were scrubbed.

The untouchable Tyson explained his reasoning as to cover the situation. It was just in case they needed to recover that data in the future. Normally, hearing that a developer is storing off live credit card numbers might raise an eyebrow or two, but with Tyson being a nice guy and all, everybody in the department believed him and let it go.

Knowing the path to hell is often paved with the very best of intentions, Steve went about and deleted Tyson’s “special file”.

Conclusion

While it may sound like there’s enough work to keep an entire department of MUMPS programmers working to fix the application sludge created by Tyson and his cohorts, nothing is going to be done about it anytime soon. To clean up the tables and remove the credit card numbers robbed Steve of two month of his life that he’ll never get back.

As a final line of defense against outsiders, the keepers of the CASTLE created enough red tape to ensure that even the simplest change would take long enough to thwart any major changes to their system. In a best case, if one was very productive, he’d find himself with weeks and weeks to kill while people try to make meetings to review his code or get back to him on requirements. Of course, the good news in all this is that Steve’s efforts just might make a bullet point on the MUMPS team’s presentation at next year’s Corporate Technology Expo.