One of the most important aspects of being a hosting company is keeping track of bandwidth usage. As a lowly UNIX admin, Jay never really knew how they did that until it had broken a year back. It turned out that bandwidth monitoring was done with a creaky, end-of-life Sun system that sat near the routers and constantly polled them, adding up the bandwidth usage, generating log files, and then mailing those up the chain to the financial people. It had been set up so long ago that the tech that did it became a senior tech, and then a leading tech, and then a chief tech, and finally, a CTO at another firm.

Obviously the tech hadn't touched the bandwidth watcher after the "senior" promotion, so the boat had been quietly floating without a captain for quite some time. Fortunately, Jay was able to go in with his mining helmet and hack at it until it worked again. But there was just one problem: the logs had lost a full day of data.

A full day of bandwidth charges translated to something between $25,000 and $50,000. Faced with telling the powers that be -- the same folks who consider instant termination a great "teaching tool" -- that he had just lost the company the equivalent of his annual salary in revenue, Jay decided to fudge things a little. He took the stats from the day prior to the loss, averaged them against the stats for the day after, and subtracted 6.3% for good measure. It was ugly and could get him in a lot of trouble if he was ever caught, but it had to be done. Jay promised himself that he'd check on the server each and every day, and even try to recommend a modern replacement.

Like New Year Resolutions, Jay's promise lasted a few weeks. There were some big changes in the company: people were laid off, policies were instituted randomly, departments were relocated, and so on. It was a several-month-long bloodbath and morale-wrecker. Through this, Jay forgot about that ancient, pesky bandwidth-watching Sun computer.

A few months later, an email had finally found its way to the UNIX-Tech mailbox: Hope this reaches the right people; we're trying to track down the "bandwidth usage" report and have no idea where to find it. We need it by day end. Can you help?

As it stood, Jay was not at work. He wasn't even in the same state. Jay was preparing, along with all of his family, for his wedding was the next day. And on top of that, all of his technician friends from work were with him, and there was no one back at the office who knew anything about the bandwidth report. Some one finally managed to get a hold of one of the techs, who then reluctantly went to Jay and told him of problem. They needed him to fix it.

Sitting in his hotel room an hour or two before his bachelor party, Jay remoted in to work and discovered that the bandwidth logger crashed again. Twenty-nine days ago.

There are certainly worse things to learn the day before your wedding, but the news that $1.45M in lost revenue is not only your responsibility, but your fault, tends to put a damper on things. Surely he would be immediately fired and even sued for professional negligence. In other words, it was the perfect way to start a marriage.

Jay could have done a number of things, but he chose a simple path: fudge. This time, he'd need to fudge twenty-nine days of bandwidth instead of a single one. It wouldn't be easy and would require a fairly complex script.

He took every spare moment he had and whipped together an algorithm that would take the last six days similar to the current one (e.g. the last six Mondays), average the data, and then subtract a random amount between six and eleven percent. The script would take those numbers, fudge them a bit more so that "bizarre" trend lines didn't show up, and generate the report for mailing. In a fit of hubris, he came up with the perfect name: fudgefactory.sh.

After a few test runs, and only two hours before the wedding ceremony, Jay kicked off the script and let it mail out the reports. He was probably sweating a little more on the altar than the usual groom, but I'm sure it wasn't too obvious. And this time, he really, really, really promised himself that he would never let the bandwidth-watcher go dead again.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!