Comment On The Fudge Factory

Slow day today eh?

Not much of a story, so a day was lost, the average bandwidth for that day of the week would be a good substitute.

Anyways, nobody pays per K anymore (well except for cell phone data plans) so who cares how much BW is used now-a-dayso?

Read the full story. The front page is rather confusing now - showing only part of a story on it.

They lost 29 days of data. The day before he wedding.

Wait, I'm confused on this one. Am I supposed to loathe Jay or his company for the big WTF?

Mildly off-topic, but has anyone else looked at the address bar?

http://thedailywtf.com/Articles/The_Fudge_Factory.aspx

or...

http://thedailywtf.com/Comments/The_Fudge_Factory.aspx

Can you do fancy routing in asp.NET, or does Alex really have separate pages for each story?

Why do you always have to hate, caveman? The world does not always exist exclusively in zeroes and ones.

Josh:

Read the full story. The front page is rather confusing now - showing only part of a story on it.

Oops, I didn't realize the site has changed. Then let me post a new WTF, there's no "Read More >>" link in the article's content!

I'd say Jay. Fabricating a month's worth of data goes beyond simply unethical.

I'm sure he's using the Windows equivalent of modRewrite to do this. /Comments/ and /Articles/ are just URL parameters, not directories. "The_Fudge_Factory" in the URL is a parameter, too, I bet.

So...

Jay doesn't do his job, routinely fakes the numbers used to bill his company's clients, and makes a half-hearted promise to maybe try to not do it so much anymore.

Are we supposed to feel bad because his boss would have fired him if he were caught, or impressed that he hacked together a humorously named script to automate his cheating (of either the company or the clients, depending on whether he over- or under-faked the traffic)?

W(hy)TF does he still have a job? He makes all us techies look bad.

OK. So, someone chose the easy route of fraud, rather than stepping up and telling his superiors that their system was flawed. Then, when the problem occured, again, he chose an even more egregious path of fraud. The only WTF here is that this tool had the cojones to submit the story.

I'd say the problem is rooted in having an opaque billing process. Even when this old machine is running, there is no way to audit or contest the results. Given that existing problem, it's not so great a digression to fudge some output.

TravisO:

Anyways, nobody pays per K anymore (well except for cell phone data plans) so who cares how much BW is used now-a-dayso?

It's been years since I've had to pay per K, but paying per hundreds-of-G or per T is not so uncommon today.

On the flip side, I now finally understand where those unexplained differences between my own bandwidth usage data and my ISP's usage reports came from!

;-)

Nah, the fudged reports are not important at all - if they were important they wouldn't be managed with a half-assed solution from eons ago.

If those bandwidth reports are really worth that amount of money to the company, then why aren't they paying a significant fraction of that to ensure they get done?

The problem here is that Jay's lack of transparency. It is our duty as tech employees to communicate status to management. It is management's duty to take responsibility for that status and make decisions based on it.
The first time the system went down, the right thing to do is tell management that there was a problem, and offer some technical solutions and let management make a decision. It is not Jay's place to take ownership of that decision.
This is all you can do! Tell them the system is broken. Tell them it needs to be fixed. If they decide not to allocate resources to fix it, guess who's fault it is the next time the system dies?

ScrumMasters will recognize this as a classic Squirrel Burger story.

Doesn't like cheaters:

Are we supposed to feel bad because his boss would have fired him if he were caught, or impressed that he hacked together a humorously named script to automate his cheating (of either the company or the clients, depending on whether he over- or under-faked the traffic)?

If he had over-faked the traffic, a customer might have complained about it, possibly triggering an investigation that would lead back to our anti-hero. By under-faking the traffic, the customers will recommend this company to their friends--when I find billing errors in my favor, I tend to speak quite glowingly of the supplier in the presence of my peers. ;-)

The only logical thing to do (if you accept the premise that you must commit fraud) is to commit fraud in a way that will get the minimum number of people's attention. This, of course, leads to the question...WTF is he submitting his story here, of all places?

How many companies have people reading this right now, going "hey, we have an ancient boat anchor of a machine monitoring our resource usage, and Wally got married a while back..."?

Zygo:

How many companies have people reading this right now, going "hey, we have an ancient boat anchor of a machine monitoring our resource usage, and Wally got married a while back..."?

Nah, its a preperation for another WTF, featuring a story of a mass-firing ot mazrried techs. ;)

Captcha: Error, yeah, 27-day-log...

Milkshake:

OK. So, someone chose the easy route of fraud, rather than stepping up and telling his superiors that their system was flawed. Then, when the problem occured, again, he chose an even more egregious path of fraud. The only WTF here is that this tool had the cojones to submit the story.

Yeah, so what about mgmt's habit of canning people for reporting bad news (hinted at)? They created the environment to begin with, so they have part of the blame.

Lionstone:

I'd say Jay. Fabricating a month's worth of data goes beyond simply unethical.

In fact, I can think of at least a couple of versions of fraid which apply to $1.54M worth of falsified data. Not to mention that at least one of them is a federal offense (assuming the bills were mailed out to customers).

Yes, this isn't meerly amusing, it's actually criminal. The WTF here is that, I think, we're supposed to sympathize with Jay's predicament. Personally, the funny part of the story comes only when Jay is marched out of the office in hand-cuffs for mail fraud.

Zygo:

By under-faking the traffic, the customers will recommend this company to their friends--when I find billing errors in my favor, I tend to speak quite glowingly of the supplier in the presence of my peers. ;-)

Jeezus, man, you are too easily impressed. Maybe some of the customers had much less actual bandwidth usage in the faked-up month than they had in the previous month. Dramatic spikes and dips in traffic are a regular part of hosting a website. Even if most of the customers ended up paying less than they would have, it's likely a fair number of customers got screwed over by our hero. If he worked at my company, and I knew he was covering up bad numbers, he'd be fired. Okay, maybe I'd wait for him to get back from his honeymoon before handing out the pink slip.

How to responsibly deal with the problem from the engineer's point of view? Report it to his boss. What's management going to do? Quietly ask the engineer to write FudgeFactory, maybe. But then the shame is on them.

-Erik

spacedman:

If those bandwidth reports are really worth that amount of money to the company, then why aren't they paying a significant fraction of that to ensure they get done?

Mainly because Josh isn't telling them that there's a problem...? at least, that's my guess.

I can see the guy faking one day's results, especially if management are keen on firing people on the spot for no reason . Morally dodgy perhaps, but certainly understandable.

But at that point, fixing that system should have been a priority. Forgetting all about it to the extent that you lose 29 days data is inexcusable.

Best way, I'd imagine, would be to silently "fix" the first day, and then report the problem saying that data was "almost" lost and the problem needs fixing urgently. Pass the buck :)

Yeah, so what about mgmt's habit of canning people for reporting bad news (hinted at)? They created the environment to begin with, so they have part of the blame.

Where was that hinted at? All I saw was that they might fire him after the first time the critical system that he ignored went down and cost them 25K-50K. Instead of making sure that the systems he was responsible for were working correctly and had proper failover and recovery in place (the sort of thing that most people would expect a tech to do), he ignored this one on the specious grounds that it had never failed before. When the expected finally happened, he went into CYA mode in the worst way.

Blaming his employer because they might have fired him for electing not to do his job in the first place is weak and in no way excuses him for the ethical lapse of faking the billing records. He then compounds it by taking absolutely no steps to correct the problem.

People like this give us all a bad name and are the reason that so many non-technical people distrust us. They look at stories like this and the people defending him and figure that we are all one bad day from using our techie mojo to lie, cheat, and steal from them.

Can't be THAT hard to rig something up to email you if there is no data coming in from the server if your company's revenue stream depend on it.

Doesn't like cheaters:

Yeah, so what about mgmt's habit of canning people for reporting bad news (hinted at)? They created the environment to begin with, so they have part of the blame.

Where was that hinted at?

I assume it was this bit:

the same folks who consider instant termination a great "teaching tool"

Zygo:

If he had over-faked the traffic, a customer might have complained about it, possibly triggering an investigation that would lead back to our anti-hero. By under-faking the traffic, the customers will recommend this company to their friends--when I find billing errors in my favor, I tend to speak quite glowingly of the supplier in the presence of my peers. ;-)

Once upon a time I made a program for keeping track of some work assignments which are done 24/7. After the system had been in use for a couple of years, I noticed a bug which causes an assignment to be billed twise, if it was started in one month and it ended the next month. So they had been billing maybe one or two items (out of some thousands) per month twise, for a couple of years, and nobody had ever noticed it.

My solution was to silenty fix the bug and not to mention it in any change logs. ^_^ Before this program, they used pen an paper for keeping track of the work they did, so I doubt they would notice even if there are a couple of items more or less...

seraphim:

Doesn't like cheaters:

Yeah, so what about mgmt's habit of canning people for reporting bad news (hinted at)? They created the environment to begin with, so they have part of the blame.

Where was that hinted at?

I assume it was this bit:

the same folks who consider instant termination a great "teaching tool"

I'm not buying it. Read the context again. That is why he fudged the data: because they might fire him if they found out the critical system he ignored went down for a day. In my mind, instant termination for that level of negligence is harsh but not unreasonable. Nowhere does the article say he might have been fired for reporting that the server was at risk or that he needed to implement a back-up or even a @#$@ ALARM to tell someone immediately when it fails. He did not do his job, and then he committed fraud to keep from being found out.

Doesn't like cheaters:

I'm not buying it. Read the context again. That is why he fudged the data: because they might fire him if they found out the critical system he ignored went down for a day. In my mind, instant termination for that level of negligence is harsh but not unreasonable. Nowhere does the article say he might have been fired for reporting that the server was at risk or that he needed to implement a back-up or even a @#$@ ALARM to tell someone immediately when it fails. He did not do his job, and then he committed fraud to keep from being found out.

I agree. This is pretty egregious. Quite frankly the biggest WTF is that he committed massive fraud and had the balls to brag about it here. I hope he gets caught next time.

Doesn't like cheaters:

I'm not buying it. Read the context again. That is why he fudged the data: because they might fire him if they found out the critical system he ignored went down for a day. In my mind, instant termination for that level of negligence is harsh but not unreasonable. Nowhere does the article say he might have been fired for reporting that the server was at risk or that he needed to implement a back-up or even a @#$@ ALARM to tell someone immediately when it fails. He did not do his job, and then he committed fraud to keep from being found out.

Context? I'm not seeing where it says the original problem was due to him. He tried to fix it, got the server back but with a loss of data ... where does it say the long-forgotten server was his responsibility alone? Indeed, there's nothing to suggest he's the sole sysadmin and as he describes himself as "As a lowly UNIX admin" not "THE lowly UNIX admin" it's entirely possible he's a member of a team of sysadmins.

Lionstone:

I'd say Jay. Fabricating a month's worth of data goes beyond simply unethical.

I agree. Even one day's fudging is unethical. Even an hour is out. 29 days... That's just... I don't know the word for this. Illegal? Sanctionable? Immoral? If Jim had mentioned the problem before, then his ass would be covered.

No matter what you've done, hiding it makes it worse. The first time it happened, he should have said, "The monitoring machine crashed and lost the day's data. We have to spend $X to fix the problem." Email it to everyone in purchasing, management, legal, and BCC a copy to your lawyer and your home account.

Then when they fire you, you can say, "they didn't want to fix a problem with a critical server." You'd even qualify for legal protection as a whistleblower.

You shouldn't fudge the data. That should be something considered anathema to anyone working professionally in any field. If you don't have it, you don't have it. They could have billed the clients for their average, and say "Due to a tracking error, you are being billed based on your average usage extrapolated using a something something algorithm. If you require more information, please contact our Department of Mathematics for a long, tedious lecture."

Jim's the villain in this story. If your company will fire you for making a mistake or pointing one out, then you should find another job. "There was pressure to be perfect" is not a valid defence for being an unethical sloppy nitwit covering his ass by scripting on his honeymoon.

One of the things that is so great about an organization that fires people for some of the most mild of blunders, including errors that occur due to bad decisions made by others--is that it forces everyone to use "subversive ethics" to keep the company running, as in this case.

In fact, when the threat of firing is used like a baby saber, everyone keeps their mouths shut about all except the most egregious problems [i.e. accounting scandal at Enron].

It only makes people hide small problems that wind up turning into big problems [and effing you even more than effing over your staff].

I have a really hard time believing that a product that is "in charge" of generating 25k-50k of charges simply get put on the back-burner like that.

While 25k-50k might not be that much, depending on what company you work for and such, but that is income.

As a lowly UNIX admin, Jay never really knew how they did that until it had broken a year back.

Except that the story starts with:

One of the most important aspects of being a hosting company is keeping track of bandwidth usage. As a lowly UNIX admin, Jay never really knew how they did that until it had broken a year back.

I took this into to mean that he had no idea about the existence of the system until it broke. IMHO, if anybody was negligent, it sounds like his employers were. I know that I certainly would love the idea of working for a company where I was held responsible for a mission-critical piece of equipment that nobody has ever bothered to inform me about.

Josh:

Mildly off-topic, but has anyone else looked at the address bar?

http://thedailywtf.com/Articles/The_Fudge_Factory.aspx

or...

http://thedailywtf.com/Comments/The_Fudge_Factory.aspx

Can you do fancy routing in asp.NET, or does Alex really have separate pages for each story?

Yes, yes you can. In the ASP.Net world we call them HttpHandlers. They are triggered by a string match in the url, such as "/Comments/" or "/Articles/".

Two different utility companies in the US within the last 12 months sent out fictitious bills, in excess of $1,000.00 per subscriber.

The phony bills were purposely sent out to deadbeat or frequently late-paying accounts to get them to contact the utility (where the utility was monitoring and capturing CallerID).

That's when the real amount in arrears was disclosed. Both incidents generated a furor, but it got the job done. Almost everyone slapped with those huge bills called the utility (when they weren't returning calls or sending in payments).

baldheadedguy:

Lionstone:

I'd say Jay. Fabricating a month's worth of data goes beyond simply unethical.

In fact, I can think of at least a couple of versions of fraid which apply to $1.54M worth of falsified data. Not to mention that at least one of them is a federal offense (assuming the bills were mailed out to customers).

Yes, this isn't meerly amusing, it's actually criminal. The WTF here is that, I think, we're supposed to sympathize with Jay's predicament. Personally, the funny part of the story comes only when Jay is marched out of the office in hand-cuffs for mail fraud.

MikeDawg:

I have a really hard time believing that a product that is "in charge" of generating 25k-50k of charges simply get put on the back-burner like that.

I find this entirely believable. I know of a company that has a creaky old homebrew application that is responsible for managing well over $100 million in annual revenue, and they're not interested in spending money to make it work better.

Did not know that. Thanks.

I didn't know you could have The_Fudge_Factory.aspx and not actually have that file exist. (I've only done stuff like that in my limited ruby on rails experience).

Heh. My spidey senses tell me that Jay's stunt will generate significant fallout. For one thing, a script hacked up in a couple of hours in a stressful situation is going to have a number of bugs in it. For another, even aside from bugs, you're going to have oddities like customers who canceled their accounts last month, who are now billed for bandwidth use during a month in which they didn't even have a server up. That's going to raise some eyebrows in the billing and the customer service departments. And then they're going to ask Jay to look into the accuracy of those reports... it's all going to come out, one way or another.

Josh:

Did not know that. Thanks.

I didn't know you could have The_Fudge_Factory.aspx and not actually have that file exist. (I've only done stuff like that in my limited ruby on rails experience).

Yeah, you could also trap HttpException in the Global.asax and internally remap the URL if the HttpErrorCode = 404, but that's an ugly hack and has the performance penalty of allowing an Exception to be thrown in the first place (one of the golden rules of most programming languages is "Exceptions are expensive")

(Captcha: 1337)

My department was once given the job of automating a process that had been previously done by some combination of Excel and Access magic. The resulting report was given to the CEO every month as a status report.

In working on the project, we discovered an error in their process that was inflating some of the numbers. The department begged us, then ORDERED us to include the same error in our program so they didn't have to explain to the CEO why he had been getting incorrect numbers the whole time.

I was thankfully moved to another project shortly thereafter. I'm not sure who won out in that particular disagreement.

I'd like to know too. Please tell me there is a better way then this 404 page hack I found here:

http://www.asp101.com/articles/wayne/extendingnamesaspx/default.asp

(If that is the "correct" way, then I apologize for calling it a hack. [Still is a hack though IMHO ;)]

Actually, it sounds to me like instant termination is a teaching tool that Jay (or, rather, Jay's successor) could have learned a lesson from. Seems like nothing sort of a firing would actually teach Jay how to properly maintain a system instead of straight-out making data up to cover his ass, considering he's done it twice now.

(captcha: enterprisey. Actually, I think enterprisey software would do a better job than Jay did.)

David Nelson:

I'd say the problem is rooted in having an opaque billing process. Even when this old machine is running, there is no way to audit or contest the results. Given that existing problem, it's not so great a digression to fudge some output.

Would you please pass on some distinguishing characteristics to go with your name, so that I can make sure no organization I'm part of hires you?

It's like saying it's okay to rape a passed out woman, as long as she's kinda slutty when she's awake.

hippie beatnik:

Why do you always have to hate, caveman? The world does not always exist exclusively in zeroes and ones.

Liar! Take it back!

I'm not seeing how this isnt plain, old fashioned, fraud.

Sure, they have a teetering old app that measures the bandwidth usage, but the accountants and company management shouldnt be expected to understand why the app needs to be replaced - thats why they have the techies in the first place. Jay should have immediately told them after the first failure.

This whole thing isnt a WTF, its a CYA. Which, imo, is pretty sad. I have no simpathy for stooges who live in fear for their jobs. Get a life and grow a pair. On your wedding night? How pathetic and weak.

CB:

The department begged us, then ORDERED us to include the same error in our program so they didn't have to explain to the CEO why he had been getting incorrect numbers the whole time.

At that point I'd be tempted to ask them how much such a purposely coded error was worth to them :-)

Nah, see, Alex changed all the details again. It's not an ISP, it's an outsourced customer-support call center. The old box is a Mac, not a Sun. The traffic it's monitoring is incoming phone calls, not network traffic. His name isn't Jay, it's Claire.

And the donkey is really Indiana.

CAPTCHA: paste. Mmmm, paste.

obediah:

It's like saying it's okay to rape a passed out woman, as long as she's kinda slutty when she's awake.

That could be the most disturbing, kinda funny, yet perfect, analogy for this situation.

Actually, he's quite right.

There *is* no auditing (obviously) of their process, or they would have discovered a problem long ago.

A better version of this bandwidth solution would have the source not only tracking the bandwidth, but it would also generate a report (or put some data element somewhere to be verified), that what is billed equals what was reported. Then further. This is analogous to collections having to match what was deposited with what applied to the G/L and A/R.

The same logic is used elsewhere. Your grocery store clerk already has her cash drawer accounted before she even receives the drawer to put in the POS machine. The POS keeps tracks of all the instruments in the drawer and how long she's working, the items she has tallied, how fast she is doing it, and what her end total is (usually called an X report). The Z report shows all the shifts that were put into the POS for that day. They should match up with what was collected in the drawers minus their beginning balances.

I/T billing systems are often devoid of any accounting controls, and instead reports like this bandwidth report are treated like the "Word of God". I bet they just take the number and make one G/L entry and a bunch of A/R entries for each hosting client and leave it at that.

John:

I'd like to know too. Please tell me there is a better way then this 404 page hack I found here:

http://www.asp101.com/articles/wayne/extendingnamesaspx/default.asp

(If that is the "correct" way, then I apologize for calling it a hack. [Still is a hack though IMHO ;)]

OMFG - Thanks for the bigger WTF than today's entry. What a load of crap, and be sure that is not the correct way to have dynamic paths in ASP.NET.... As with everything, there's a few different ways to achieve the same result, but usually I'd do this with a simple HttpHandler. Simple object, few lines of code, a little configuration and it's done.... What Alex is doing could be even simpler, just a little configuration in IIS.

-Me