Comment On Secured Debugging

In 1968, when David Foskey finally had the opportunity to stand face-to-face with a Honeywell 8200, his expression was nothing short of awestruck. Technically, the 8200 didn’t have a face, it had a window that overlooked a room. And not just any room, but a room the size of a freakin' basketball court. In fact, Honeywell recommended that no less than six thousand square feet be dedicated to the 8200; any less and the sorters, collators, processors, storage devices, and computer operators would be a bit too cramped.

For David, his enthrallment wasn’t just about the sheer size – to reiterate, a freakin' basketball court – or the mainframe’s raw computer power – over 400,000 operations per second and 1,048,576 bytes¹ of memory. It was also the fact that a Honeywell 8200² played a memorable role as war planning, Latvian army controlling, deviously plotting supercomputer in a recent Harry Palmer spy movie, Billion Dollar Brain. And not only that, but the Honeywell 8200 that David peered into was owned and operated by the Department of Defense.

Snapping Out of It

“Okay freshy,” the jarring voice belonging to David’s new boss bellowed, “this officially concludes our little tour.” His boss abruptly drew the curtain on the window overlooking the Honeywell 8200 and quietly added, “now don’t tell anyone I showed you the 8200…. unsecured.”

David wasn’t exactly sure what “unsecured” meant, but he was used to hearing it. As a recently hired software support engineer for the Department of Defense, he had to jump through all sorts of hoops just to get the “peon” clearance level. At that level, he was granted access to work at his desk and, after following proper sign-out procedures, use the restroom.

Just about everything else – even telling another peon your telephone extension number – was expressly forbidden. After all, if you knew enough phone extensions, you could better guess the number of people in the building and gauge its communication capacity. And in those trying cold war times, anyone could be enemy.

Day to Day

As a software support engineer, David’s day-to-day responsibilities weren’t all that different than what they’d be today. The Department of Defense had many large systems and programs, and like any large custom-built software, it was rife with bugs and “features”. When a program crashed or produced unexpected output, David would analyze the core dump, read the code, look at the data, and debug it until it worked.

At least, that was the “best case scenario”. For security reasons, software support engineers weren’t allowed to see actual live data. Nor did they have access to all of the program code. And core dumps… since they might contain sensitive data, they were scrubbed before ever being printed out. But aside from that, programmers could debug to their hearts’ content. Well, so as long as they did so with paper and pencil. And they shredded everything afterwards.

It took a little while, but David got used to it. He was among eight other software engineers, all of whom had similar peon security clearance. For his first several weeks, David worked at his desk and never had a chance to lay eyes on the majestic Honeywell 8200. That is, until he experienced the first development emergency.

A Serious Emergency

”Listen up, freshies,” David’s boss announced one morning, “we have a serious emergency with ALDV-IN-8862! This has to be addressed now, so drop everything and prepare to work on this.”

For the DoD, you’d think that a “serious emergency” would involve armed conflict, hostile state actions, or at the very least, a death or a maiming. But in this case, the issue was that ALDV-IN-8862 was crashing halfway through an automated inventory reconciliation. As boring as it sounded, fixing it would be incredibly exciting because it’d mean a trip to the Honeywell 8200 and the opportunity to work on some real, live programs.

A few hours later, David’s boss asked for few a volunteers to visit the 8200. To his surprise, David actually volunteered; a few others were volunteered by drawing the short straws. On the brisk walk to the 8200, David’s boss casually mentioned “by the time we get there, they should be done securing the room.”

With everything seemingly secure and secured, David wasn’t sure what meant and decided to wait to see for himself. When they arrived, David’s boss peeled back the curtain on the overlooking window. “They’re about done,” he said, “I’m sure they’ll let us in any minute now.”

David snuck a glance through the window and quickly realized what “secured” meant. The computer operators who were normally responsible for mounting and unmounting tapes were running around with handcarts and dollies, moving hard drives and other refrigerator-sized devices out of the room.

Fifteen minutes later, David and his team were permitted in the room and were told to go to work. The room was notably emptier, but none of that mattered: David would finally get the chance to sit down and debug a Honeywell 8200. As he sat at the console, the operators handed him and each of the other programmers an inch-thick stack of green-bar paper.

On the paper was printed a core dump; like other core dumps they received, there was a fair amount of sensitive information cut out. And, like other times, it was literally cut out: the operator had taken a pair of scissors and snipped wherever a sensitive buffer was printed.

Six hours later, the programmers came to a conclusion: there is no way the software is broken, it must be bad data coming from one of the I/O devices. They recommended that the hardware engineers start their diagnostics. Of course, for security reasons, the groups weren’t allowed to interact, so the room had to once again be secured for the hardware team.

After a few days and endless core dump printouts, David’s team was absolutely certain it was a hardware problem and the hardware team was absolutely certain it was a software problem. Fortunately for both, the problem went away… which meant it was probably a date problem, somewhere. As the days turned to weeks, weeks turned to months, and the pile of unsolved core dumps grew to be tens of thousands of pages, David came to realize something. The Honeywell 8200 wasn’t all that impressive anymore. He even felt relieved when he drew the long straw and wasn’t volunteered to go visit the mainframe.

[1] Technically, the 8200 (and the earlier 200) series mainframes used 6-bit “characters” instead of 8-bit bytes; so that’d be 6,291,456 usable bits, which would be the equivalent of 786,432 bytes. Kinda. Go read the manual if you want to learn more.

[2] Okay, while we’re doing footnotes, it was the technically the 200 that starred in Billion Dollar Brain; but they were basically the same thing, except that the 8200 was even more awesome.