Comment On The Cost of High Security

A few years back, I was “between jobs” (voluntarily, for the record!) and thought I’d try one of those generic “.NET developer needed for three-month contract” jobs. As I learned at the interview, the client was a Certain Federal Agency, and this Certain Federal Agency required that all contractors have clearance at the “Confidential” level. But not to worry: although my then-clearance level was at the “Hoi Polloi” level, the Certain Federal Agency was happy to bring people on board now (as they needed people “right away”) and have them “wait a little bit” while the clearance request went through. Sounded reasonable enough to me, so I took the job.

Three months later, having accomplished little more than mastering the Rubik’s cube and memorizing the forty-eight pages of “non-sensitive” documentation, I was asked to extend my contract another three months. My security clearance (which would allow me, among other things, to log in to the domain and finally see the core application), was “right around the corner.” I hesitantly agreed and signed up for another three months.

Another three months later, I had two more major accomplishments under my belt. Not only could I solve the Ultimate 5x5x5 Ruibik’s Cube blindfolded with my feet, but I had translated the forty-eight page “non-sensitive” documentation into a regional dialect of Esperanto (which I also had invented). Yeah, it was a little slow. But the good news was my security clearance had come in. Well, almost: near the end of the sixth month, I was finally granted read-only access to the codebase. Yet despite all that progress, I declined another contract renewal. Apparently, it could take up to a full year to gain commit access to codebase and read-only access to the production application.

In hindsight, I don’t consider my experience to be a “WTF.” Although the application didn’t contain defense secrets, the data were sensitive and, in the hands of a nefarious individual, could be used to wreak all sorts of financial havoc on millions of individuals. It would be a WTF not to perform an extensive background investigation in this manner prior to handing out that information.

Sure, they could have only hired developers with clearance, or even required that developers wait until clearance, but we all know how hard it is to find a qualified developer. Let alone a qualified developer with clearance or even a qualified developer willing to wait, jobless for months while clearance came in.

Sure, they could have given me read-only access to the source code on Day 1, but had I been that nefarious individual, I might have found a vulnerability in the system and used that to attain sensitive production data. And sure, the system developers could have performed an extensive security review of the system to ensure that developers can’t hack in, but they simply didn’t have the resources to do that. How could they with it taking at least six months to give a new hire read-only access?

As for speeding up the background investigation, that’s not too easy, either. Sure, they could hire more investigators, but just imagine how extensive of a background investigation would be needed for a background investigator. And with every agency demanding background investigations due to our “heightened state of alert,” how could investigators find the time to bring more on more of their own?

In the end, it all comes down to down to the Cardinal Rule of Security: a security system is only as strong as its weakest vulnerability. As citizens, we demand that our government (especially the Certain Federal Agency) protects our sensitive information with the utmost care. And as taxpayers, we pay a lot for that level of security. That’s just the cost of high security.

On the other hand, a lot of people want that level security, try to implement it on the cheap, and then are shocked when it either costs a ridiculous sum of money or fails miserably. One company that found itself in this situation was a large, well-known international company that “Steven B” once worked at:

Not too long ago, I interviewed at CyberCorp (as I’ll call them) for a contract developer position. Through the four-week interviewing process, they assured me that I’d be able to start immediately. This was important to both of us, as I was out of work and they had a huge developer shortage. Their offer letter, however, told a different story: my start date was a full six weeks away.

Try as I might, they refused to push the start date closer. Apparently, they had to “prepare for my arrival” and perform “background checks” just to make sure my story checked out. I wasn’t too thrilled to have another unpaid six weeks, but agreed and stuck around.

When I finally arrived on site, I logged on to my workstation only to find a few roadblocks. First and foremost, there were no developer tools installed. Secondly, I did not have Local Administrative rights to install these tools. Not that it mattered, of course, because the CyberCorp’s “Security Policy” explicitly forbade the installation of any software on its machine.

This was to be expected, so my manager said. She pointed me to their web-based support system where I could open a ticket and request permission to install/configure developer tools on my workstation. Five days later, after constantly pestering the IT Department, they finally responded: “we’re investigating the issue.” This was also to be expected, so my manager said.

The following week’s staff meeting, however, told a different story: the CTO lambasted “us contractors” for “just sitting around and making excuses” instead of “actually getting work done.” My manager mentioned that the company’s new security policies were preventing the new contractors from writing code. “Unacceptable,” said the CTO, “there’s a lot to do other than write code.”

A few days of “pre-coding” in notepad later, I noticed that my ticket requesting access had been closed “without resolution.” My manager said this was to be expected. So I opened another requesting the same thing. That was also closed “without resolution.” After some minor teeth pulling, I finally was able to speak to an IT person on the phone. Apparently, for security purposes, contractors could not be given this access.

So I opened another ticket requesting that the IT Department install/configure development tools. They replied back with an estimated day of completion: two and a half weeks from that day. I continued to do more “pre-coding” in notepad.

A week or so later, while still waiting for the development tools to be installed, the CTO held another staff meeting. There was a “major organizational change” and they would be prematurely terminating all on-site contractors. Apparently, the bill for all the contractors was “way too high” and “entirely unnecessary.”

Fortunately, my manager and several other development managers were able to plead with the CTO to keep the contractors on board for another few weeks. Seeing that we made up nearly two-thirds of all development staff, such a reduction would have seriously put everyone’s project in jeopardy. Not only that, but most of the contractors were still waiting on the IT Department to configure/install their workstation: without local admin rights, they couldn’t even debug code.

Finally, after another few weeks, the IT Department installed Visual Studio on my workstation. As happy as I was to finally be able to write code, the announcement from earlier that day overshadowed my delight. The powers that be were “fed up” with “the giant contractor bills and minimal productivity” and announced that they would be prematurely terminating all on-site contractors. That was to be our last day.