Dave sat in the conference room with his supervisor, Chilton. Every other participant of this meeting, supervisors of remote teams, looked on from a subdivided, giant projector screen. Each remote office was in a different time of day.

"Release 2.12 of TaskMaster was successfully pushed out this morning, UK time," Chilton said. "So far the user feedback has been above average. The new CSS transparent menus were a great addition from your team, Dave. Well done."

Dave nodded modestly.

The rest of the meeting was a parade of accolades for yet another "triumphant" release, per Chilton's habit. About forty-five minutes in, Dave's phone buzzed. Someone from QA had filed a bug: Transparent Menus are completely invisible in Internet Explorer 8.

I thought we tested for regressions, Dave thought. After excusing himself from the meeting of accolades, Dave hurried to Sally's desk, who was the local head of QA. "We turned off one of our regression tests by accident before launch. The CSS menus don't even appear when you click on a menu item."

"It'll rile up Chilton something awful, but we'll have to roll back to 2.11." There were a dozen high-priority clients who still used IE8 in corporate environments, so they had to support it. "Get me the last revision."

Sally pulled a list from the repo. The last stable release was 2.11.20120101194553. "I'll start the rollback process." She groaned. "So much for lunch … wait. The build isn't on our server in Boston. We can't roll back to that revision."

"So? Do the one before that."

"I can't. None of the revisions in the repo actually exist on our Boston server."

The Git is Afoot

"Yes, this is Dave from TaskMaster. Connect me with your sysadmin, please." Dave was on hold for the fortieth time that afternoon. He had tried desperately to find a sysadmin who hadn't yet upgraded to 2.12 and who could tell him what version of 2.11 they had installed. He was now trying some textile ship in Brisbane, Australia.

"This is Pavel," a voice grumbled.

Dave introduced himself. "I need you to do me a favor. Tell me what build you're using."

Pavel groaned. "Of course, mate." He muttered something that Dave could never repeat and typed for a few minutes. "It's on build 2.11.20120101094553."

094553? I thought the last build was 194553. "Thanks, Pavel."

Dave hurried to Sally's desk. "Do you have 194553?"

"One moment." Sally looked through the repo. "No, no such build."

"Well, that's what's installed in Australia."

"Well, that's impossible," Sally said. "Unless someone changed the build number on their end."

"Hold that thought," Dave said. He reconnected to Pavel, the drowsy sysadmin from Brisbane. "Pavel, what upgrade process do you use there?"

"Well, we download the patch from your servers, run the setup scripts locally, and it's done."

"Thanks." That's the way it's supposed to be done, he thought. So why would there be an unknown build number on their system? We do the same thing here, and the only difference is--

194553. 094553. Timestamps.

Dave uttered something very impolite and paid the build team a visit.

Elementary, My Dear Subversion

"Right, well, we append the Unix timestamp at the end of every build version," Jason said. He was the head of the build team at TaskMaster. "It's easiest to keep track of all those point releases."

"Why don't you double-check?"

Jason grumbled, but pulled up the source code. "Right, here, see, this is where -- oh, now isn't that a surprise." Jason pointed to a line. The timestamp was being generated with localized date formatting. "That could be a problem."

Dave explained everything to Chilton later, after they pushed out a revision of 2.12. "We tried to roll back to fix the menu issue. But the team had been using local timestamps on all their builds, so clients in Boston, Brisbane, Budapest, they were all using differently-numbered versions. We pushed through the newer timestamps with the fixed transparency issue."

"Commendable," Chilton said. "Will you stay a few minutes? I'd like to get the others on the projector."

"No, that's not necessary," Dave said hurriedly. "I have to see to something."

Dave went back to his desk. He knew better than to sit in on another accolade meeting. Catastrophes tended to happen.