J Cooper sat down with his buddy Holden for lunch. They'd eat together pretty much daily, enjoying a meal while discussing work. While they were eating, Holden mentioned a particularly irritating issue that he was troubleshooting.

Holden explained the process as it should work — clients would submit flat files which would automatically be read into the PICK database, then invoices would be generated. If there was an import error, it'd send an email alert to the representative that handled that particular client.

It wasn't uncommon for the import to fail, which made this a major issue. If the reps weren't getting the emails, they'd find out about incorrect invoices from irate clients. If the clients were irate, that meant they'd already spent time researching invoices when they could've been working.

"So why are you here, eating lunch with me, when we're wasting our time and our clients' time?" asked J.

"Because it's not our team's issue. I'm getting to that," replied Holden.

Holden went on to describe his troubleshooting process. He cloned the production environment for testing, assigned himself as the rep for a client with a bad data file, and submitted the file. In the time it took him to look down at his watch, a new email message appeared in his inbox. Sure enough, it was the notification email that'd be sent to the reps when the import failed. He tried the same thing with a dummy client in production, and received an email in seconds.

"So it's a mail server issue?" J asked.

"Yeah, I think so. I signed off on my testing and assigned it to your team. Holden was on the dev team, and J was in network engineering.

J excused himself and went back to his desk to find an email asking him to investigate "mail server problems." He read the issue description, and it was just as Holden had described. J did everything he could think of; sending messages directly via telnet, emailing inside and outside the organization, cloning the exact message text and body, and couldn't reproduce the issue. During their daily lunch, J told Holden his findings.

"I don't know, Holden, I can't figure it out. I've tried everything; it has to be a code problem," J explained.

"Come on, I've tested it at least a dozen times and not once has the email failed. This is your team's problem," Holden insisted.

"Holden, I'm telling you, I've tested this every possible way that the email could fail. The mail server's configuration hasn't changed in over a month, and this only became a problem like a week ago. I'm assigning it back to your team; I'm sorry." Neither J nor Holden liked the tension that was building.

Two months passed with them playing inverse tug-of-war, each insisting that the issue was the other's problem. Their friendship was deteriorating to the point that they were dreading lunch together, though neither wanted to admit it. Meanwhile, the invoicing system was still a wreck.

The next time they sat down for lunch, Holden seemed strangely quiet and reserved. J asked if anything was wrong, to which Holden answered "I found the invoicing email problem. It's fixed now." Enjoying the sweet, tangy nectar of vindication, J asked what the problem was, trying not to smile. "I fixed it, it's irrelevant."

"Come on, Holden, tell me!" J insisted. After some pressing, Holden finally opened up.

"Well, I was working late one night to debug a different issue, and I'd forgotten about a little piece of code I put in." Not coincidentally, it was more or less exactly when the problem started.

// TODO: remove debugging code
if (toAddress == "[email protected]")
	SendAlert(toAddress, alertText);

Lunch was on Holden for the rest of the week.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!