Computer-security-incident-initial-process.png

Git is a divisive piece of technology. There's a number of people who insist that it's the best of all possible version controls, often citing the fact that a complete repo copy is on everyone's computers in case of emergency. There are also a lot of horror stories of people screwing up commands and ending up neck-deep in tutorials, desperately trying to undo what they did. Recently, I was involved in a discussion about the merits of Mercurial. The usual git fans stopped by to ridicule the lack of history-rewriting in Mercurial, insisting that it's a necessary part of any version control. Which reminded me of this reader submission ...

Toni worked at a certain company that worked on inventory systems, and was also a defense contractor. Her manager quit out of protest; the new boss—whose name was Alexander, but he insisted on being called "Lex"—was assigned to the team from upon high. Lex was a dopey corporate puppet who liked to talk about Bob Dole a lot. The best the team could hope for was that he'd stay out of their way, making vague promises about the future but not actually accomplishing anything. After all, they had done just fine without a boss for eight months.

What they didn't know? Lex had been given the keys to the kingdom: their server git repository, and everything in it.

One fateful morning, Toni woke up at 4:00 AM to an SMS saying that the code repository at work had been accessed by "a new member of the compliance team."

... What? she wondered, groggily. There was no new member on the "compliance team." That wasn't even a real department name! Damn it, we've been hacked.

She rushed to the office, wide awake at this point, praying for no cops with speed guns on the way there.

Did someone break in? she wondered. Or worse, hack into the local code repository from one of the ports the IT guys kept forgetting to close? Surely a randomly generated password longer than a bible couldn't have been cracked, right?

By 8:00 AM, she hadn't found any sign of intrusion. She was asleep at her desk from exhaustion when her co-worker, Clarissa, came in. Clarissa had just wiped her MacBook Pro clean the night before due to a botched OS X update. She had also gotten a text message, but had assumed it was Toni, as she'd been working late the night before.

Clarissa was able to discover what Toni had overlooked: to their horror, two new people, with names 20 characters long, had been "invited" into the team, and the team had been "renamed" to Compliance.

That was when they both got an email from Lex. I thought you could use some help! :)

That "help" came in the form of two dimwits from the Chennai, India branch, recently acquired by merger a year before. Through Slack, they admitted to the existing team members that they had never used a version control system before.

Toni was already seeing red at this point, but the day was just beginning to reveal the depths of insanity that were in store for them.

The two new team members had decided that the color of the web-based inventory front-end wasn't "good enough." Now, this had just gone through a redesign; the colors matched the official design guideline, and had been agreed on by numerous stakeholders. Not only did the newbies change the colors, they decided to deploy those changes to the main Internet-facing servers without going through QA.

Toni was still processing that little gem when she heard the fires of hell erupt from Clarissa's cubicle.

Clarissa told Toni that their "friends" had pushed code to master, bypassing QA, while also tinkering with her custom branches. As if that wasn't bad enough, they'd used an editor that had converted all the source code files on her branch to Windows format, which screwed up all the line endings. Immediately, this caused conflicts with the existing frameworks, and one of their parsers no longer worked.

In a fit of brillance, the two new helpers had decided a force push would do the trick. Surely it couldn't be their color tweak breaking things, right? Apparently, they'd figured out that it was. Shortly after, they'd copied over the files from master, pasted them in the working folder, and force pushed with a fake name, erasing all history on said branch in a terrible attempt to cover their tracks.

Remember how Clarissa had wiped her MacBook the night before? If Toni hadn't cloned the entire repository to her ThinkPad for Thanksgiving tinkering, Clarissa would've lost four to eight months of work.

They received word that the interim manager, on vacation in Tampa, had hopped on a plane after getting the same SMS they did. He could've flown in with his own two arms given the level of pissed-off he was when he slammed open the office door. Lex walked in with coffee to see the door nearly fly off the hinges.

It turned out that the website had been inoperable for eight hours. With worldwide clients, this was a bad, bad thing. After the interim manager finished steamrolling Lex with expletives, he told the team that, due to US regulations, using foreign workers without security clearances could bring up to $10K of fines per infraction.

Salespeople barged in behind him with questions about why the interface had changed without any notification. The QA team followed the salespeople, also demanding answers.

Lex turned the color of wax paper. "Well, I guess I should remove the other twenty guys I just added to the repository, then."

Lex is no longer with the company.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!