David pulled his headphones off when he heard a loud harrumph behind him. One of his project managers loomed in the doorway, and had obviously been standing there for some time, trying to get David’s attention.

“You pulled from Staging-Core branch into the Version–2 branch and broke Liam’s changes,” the PM said.

David wracked his brain, trying to remember this particular PM’s name. He’d met so many during his short time at this company that he couldn’t keep them straight. “Uhh… we just had a release, right? Since I was working in Version–2, I pulled the latest version in. I thought I was the only one in there on this project right now…”

The PM shook his head. “I don’t know how you did things at your old job, but here we’ve got policies and procedures. This isn’t some tiny startup- we’ve got a dozen developers in this office, fifty more in India, another dozen in Mexico, and nearly a hundred testers. I’ll send you a meeting invite so we can discuss…”

The PM wandered off, mid-sentence, but was instantly replaced by another. This one David recognized- Lisa or Linda or Lindsey or something. “Liam tells me you’re breaking branches in our code control system,” she said. “I’m not sure how things worked at your old job, but this isn’t some tiny startup. Our codebase is over 1.2 million lines of code. I’ll send you a meeting invite so we can discuss…”

A visual report of the branching structure of the codebase in question. It's over twenty branches, with relationships that make no sense at all, like having Version 2 be a child of the Version 1 branch, and two different staging branches- one the parent of Production, and another the Child

Over the next quarter of an hour, David was visited by the ghosts of project management past, present and future, each with dire comments about David’s ability to follow policy and the promise of a follow-up meeting. By mid-afternoon, his calendar for the next week was filled with meetings.

A normal organization might do work in individual feature branches and when a feature was complete, they’d migrate those changes back to the root or trunk branch for release. People working in other feature branches should, at some point, pull those changes down into those branches. When and exactly how that happened was a possible topic of debate, but David’s new workplace wasn’t interested in a debate.

“You simply don’t pull changes down into lower branches. Ever.”

David sat at a conference table, surrounded by project managers he didn’t recognize.

“That’s why we have a Changes Under Merge team,” one of the PMs explained. “It’s for safety. Changes made by developers stay in one branch, and the Changes Under Merge team move them back up the tree, following the Change and Release Approach Policy.”

If David needed changes synced between branches, he needed to work with the Changes Under Merge team, which was a four-person group who understood how the 20+ branches in their codebase related to each other. If, for example, David needed a feature merged in, he’d need to look up all of the changesets that were used to create that feature. He’d then send them- as an Excel spreadsheet- to the Changes Under Merge team. Someone from that team would then cherry-pick just those changes and use TFS’s “baseless merge” tool to pull those changes into his branch.

Conversely, when he had a feature ready to go, he wouldn’t merge it up the tree. He’d compile a list of changesets and pass them off to the Changes Under Merge team, which would then pull the changes from his branch into Integ-Core, for integration testing by QA (there were no automated tests, because, as one of the PMs put it, “Automated tests only test your automation. You need real humans for real tests.”). Once the feature passed Integ-Core, the QA folks would request that the changes get merged into Strategic-CombRes, using the same cherry-pick and baseless merge approach. From there, the “finished” feature entered a twisty labyrinth of project management processes that David didn’t need to worry about. His feature would hop around the various Strategic Branches for awhile, and someday, maybe, make it back up into the Production branch.

David took his scoldings, did his best not to roll his eyes, and got back to work when the project management machine decided that he’d been punished enough. For the next six months, he basically ignored everything he knew about branches, and just rigorously tracked his changesets so that he could pass them off to the Changes Under Merge team. Eventually, he started to hear whispers- a major release was coming, and everyone’d need to put in some extra time to make sure it went off without a hitch.

Lisa/Linda/Lindsey descended on his cube on the eve of the release. “Now, since this is your first major milestone,” she said, “I wanted to stop by and review what’s going to happen. You’re going to get a list of changesets from the Changes Under Merge team. You need to look at the latest version of the Production branch and verify that all of your changes are in there. And no, you can’t just look and see if the changeset IDs are in there, because sometimes changes get overwritten by later changes. You’ll need to manually verify every line you’ve written. You’ve got two hours.”

This was six months worth of work. David didn’t even know where to begin with “verifying every line” he’d written. And in only two hours? It was impossible. Instead, he focused on spot-checking the release while wishing he’d been allowed to write some automated tests around his features. So far as he could tell, everything looked okay, so he signed off on the release.

The release crashed and burned. Since the company didn’t have a backout plan, and since their branching structure was convoluted, they couldn’t simply pull the previous version out of source control and redeploy it. Instead, every developer pulled the latest version of the Production branch and worked until 2AM trying to patch all of the problems in it.

There were a lot, and when the blamestorming session finally came around, several project managers were pointing their fingers at him. “You said all of your changes were in the final product, but it looks like Liam’s changes overwrote some of your changes.” “We need people that are going to follow policy.” “We need people who show real diligence.” “We don’t think this organization is the right fit for you.”

It wasn’t until he was 45 minutes into the meeting that David realized he was being fired. He wasn’t entirely certain that all of the PMs realized it either, because a few of them kept the meeting running. “Obviously,” one of them said, “while David was at fault, this problem is also a management problem. We need to expand the management team so that we can avoid these problems in the future. And since our headcount just shrank…”

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!