- Feature Articles
- CodeSOD
- Error'd
-
Forums
-
Other Articles
- Random Article
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
The point is that the billing process is opaque and there's no way to validate or verify any of the bills. It's a real-time system and if it stops working, there's no way to ever figure out what happened during that time period. So the blame doesn't fall squarely on the code monkeys when the antique system takes a dump. Granted it's not a good idea to make up data, but I bet you could get some accurate estimates based on past data. It certainly doesn't warrant a comparison to rape. Poor, sensationalist metaphor.
Admin
the better way is to use Apache
Admin
Uh, are you saying that's not ok? Oops...
Admin
I'm interested to know why the Sun box mentioned has to "constantly poll them [routers]". Could it not just ask for "hey how many bytes since <date>" at the end of every day?
Isn't this what things like SNMP is for? (Not retorical, I'm genuinely interested)
Captcha: shizzle. my nizzle.
Admin
I've worked on this kind of system, and had to deliver bad news (on the other hand, I've never let the news get that bad). Yeah, I was worried I might get held accountable for something I inherited, but come on, if you can work on this kind of system, there's always another job out there for you. This guy is a criminal - that's much worse than being fired or divorced. Him & his company should be prosecuted (I'll leave the matter of whether the company can pin it all on him to the lawyers), and I would not be comfortable being told this story firsthand.
Admin
Yes, you can indeed do fancy routing with asp.NET :) A little bit of regex and Context.RewritePath will do the job
Admin
Depending on the size of the counters and the bandwidth of their connection out of the house, the counters will normally overflow way to often. Therefore there are just a very short history retained in the routers themselves.
Admin
In that situation though the utility companies know what the bill SHOULD be. They know exactly what the bill is and are able to correct it; the outrageous overbill is just to get deadbeat customers to call in. Even then, the deadbeats will not be charged what the fake bill said, nor were the fake bills generated in order to replace actual bills. What's Jay going to do when ABC company calls and complains about receiving a full-month's worth of usage after cancelling their account on the 5th of the month?
Admin
I would have to agree with everyone else that this is just plain fraud. I don't care if they fire me step #1 when I notice an issue as big as this, especially if it is my fault but even if it wasn't, is tell management.
I can't sympathize even if they did create an environment of fear. I would rather not have a job than reduce my moral standards to the point where I am stealing, because he is definitely stealing from somebody (either the company or the customers, probably both depending on the bill).
Admin
Aargh! Please put the entire article onthe front page. I hate this "click to read more" stuff.
Admin
Agreed. It would only be pure luck if he didn't get found out immediately from customers complaining that they got billed for service after their service had been canceled.
This is the exact reason why you don't fudge the numbers the first time. If you go to your boss and say, "Hey our bandwidth billing architecture is dependent on an over matched sun box, and when I went to check the problem today, the damn thing had lost a days worth of data!"
Yea, he may can your ass, but that could be explained away in a subsequent job interview as a classic case of "shoot the messenger". But when it comes down the pipe months later that the damn thing died and you lost a WHOLE MONTH because of a problem that you were perfectly aware of, and chose not to bother informing anyone about? You'll be lucky if they only fire you.
Admin
The audit servers should be necessary for the system to function. That is the only possible way to ensure an accurate audit. Why the job wasn't running on the routers, I have no idea...Even if the machine was new, and the audit code flawless, putting it on a separate machine opens you up to this exact problem.
Admin
While it is true this is Jay's FUBAR, it is also true that it is directly caused by the trigger-happy management. Thing is, I've seen stuff like this happen in other places, except it happens in data that doesn't translate into a customer bill.
Fudging data to cover up your ass is fairly common in a hostile working environment. And an employer will usually filter out people fired for "making us lose revenue" WITHOUT even asking why that happened.
Then again, I would have reported upper mgt about the server failing after the 1 day fudge. If they refuse to upgrade the system, I would then quit and search another job (so when the system crashes again, I will not be held responsible :)
Admin
Well, billing will check their records and confirm that they [should have??] cancelled the account, cancel the bill, appologise to the customer, and, not wanting to to experience the ID teaching device, keep quiet about it!
CAPTCHA: BillGates probably would have done this as well.
Admin
Looks like it ended up being fraudulent billing.
So given the mining mission when the system failed, Jay chose to mis-report what he found because he was too afraid of getting fired. (Who would even want to keep that job?)
Given that Jay had a solution for how to bill for that missing day, he really should have reported what he found and how to bill for that day. With all the changes that were to come, he might have ended up being a lead tech, and then a chief tech, and finally, a CTO at another firm just in time for his wedding.
Admin
Admin
In my first computer-related job, we routinely fudged data that was lost, garbled, or that we couldn't determine by other means. We kept records of what we did, unlike the Fudge Fellow, but I don't think anyone ever looked at them.
The main difference between my situation and the Fudge Fellow's was that I wasn't working for web hosting company. I worked for a bank.
-Shirley
Admin
I never thought I would say this, but after reading the comments... you people are not cynical enough.
You're getting all self-righteous about Jay faking these numbers, and saying that he shouldn't have done that and should have let management know.
Okay, he lets management know that they don't have the data to charge the users the agreed-upon rate. What is management going to do (AFTER they fire Jay)?
They're going to charge the users an amount based on an average of that user's typical historical usage.
It's the only thing that CAN be done when that data is missing, guys - the company can't feasibly not charge anything, but they also can't charge a random number.
Jay did exactly what would have ended up happening anyway - again, it was the only thing that COULD be done. He just did it in a way that prevented himself from being fired on the day before his wedding. I certainly can't condemn that, and I'm surprised that you people can delude yourselves into thinking you would have done anything different.
Admin
-Erik
Admin
Why disturbing? :D
Admin
I'm sick of this mentality. Hate them both, its not an either/or case here. You don't always have to place the blame solely on one entity.
If you get hit by a falling piano (and live) and later find out that the piano company used poor cabling, you you blame the people who sold the cable, or the piano mover that didn't check it, or yourself, for walking under a damn piano?
Admin
Damn, that last message is for the guy above your post. Kinda takes the wind out of that argument (and ironically, can only blame myself for).
Admin
If it's possible to put routing and auditing on one box that should be the perfect solution, yes.
If it's not possible for whatever reason, I, thinking as a customer, would be more happy with a solution where the system continues to work without the audit service -- assuming that the company does NOT send me "best guess" bills of course.
I wouldn't be very comfortable with the idea that the company stops service because some auditing service failed and they wouldn't be able to charge me for the time the audit was not running. It'd make me wonder how often that service really fails, because if it's rock solid and does not usually fail ... it shouldn't matter whether they can or can not charge me for that time.
That of course assuming that I as a customer knew about the "stop service when audit fails" policy :)
Admin
No, they're going to use the other redundant system that they didn't tell Jay about either.
Admin
If they refuse to upgrade the system, I would then quit and search another job (so when the system crashes again, I will not be held responsible :)
Definitely the wrong order to do that in. Never quit 'til you have another job lined up.
Admin
One WTF that hasn't been mentioned is why the company took 29 days to realise that the (supposedly) critical billing information wasn't coming through.
It's like the old story of printing and not distributing reports -if nobody has come looking for their report by the next day (or whatever the cycle is), then don't print any more!
Admin
I'd have to agree... so long as the reason for his termination is the fact that he lied and deceived management and fabricated data to do so. Sacking people for making a mistake, or worse, for not doing something they didn't even know they had to do, is NOT OK - but neither is covering up and hoping it'll go away. The only ethical thing to do would have been to have said to management "I'm really sorry, we've lost a day of bandwidth reports - here are my best estimates for what they should have been; and we REALLY need to overhaul the system so it doesn't happen again". I've been there, I know this - if you come clean at the earliest possibility, chances are you'll be "punished" by being charged with fixing it. ;) Depending on the scale of mistake, you might have to endure a bollocking - but anything worse and it's time to dust off the CV.
But to have a mistake happen, cover it up with fake data, not take steps to ensure there's no way it can possibly happen again, and then to have the same mistake happen, on a much larger scale, at a time when you can't easily fix it... I think the applicable words are "gross misconduct", with a rider of "gross negligence" to boot. As far as I'm concerned, Jay isn't someone I'd want to deal with in any capacity.
(This tale reminds me of nothing so much of the guy in the parable who hid his one gold piece in a hole in the ground because he was scared of his master coming down on him for losing it...)
Admin
Totally agree. Maybe the WTF here is management and fostering an environment that compels people to do such things.
Jay should have told his manager after the first incident and recommended solutions. If they fired him for that, then odds are he's better off working elsewhere.
Admin
WTF: He didn't know about a supposed mission critial process/server. WTF: As a lowly tech he fudges a days worth of critial data, doesn't tell anyone. WTF: No one notices 29 days of critical data is missing (or it takes them 29 days to figure it out) WTF: The lowly tech is getting married, but he is the only person in the company that can solve the problem? If it was me, I'd say you've waited 29 days, you can wait a bit longer
There are plenty of problems and issues here, but most of it stems from J fudging data, then hiding it. If he was such a lowly tech he was making a decision that wasn't his to make. And he should have informed his superiors there might be a further problem. IF he was a manager, then it is his responsibility to fix the issue for the future, which he didn't do.
Admin
Differences:
Also, aside from a one-line throwaway parenthetical, why do we believe the management of the company would have fired Jay after the first failure (which, after all, he was called in to figure out after it had happened)? Either we're missing a huge chunk of the story supporting this parenthetical, or it's just Jay's assertion. And, might I add, Jay is far from trustworthy here.
If he'd really be fired for showing a failure of a pre-existing process, then he should give that company a good riddance and move on to a place with competent management. That would have made a great WTF. IMHO, the "real WTF" here is completely presumed: that Jay would have been fired for something that was completely not his fault. But, why did he assume that would happen? Without some anecdotes backing up that single line, I can't even imagine taking his side here. Sounds more like someone justifying their failure instead of explaining anything.
Firing Jay after the second failure: I honestly can't imagine much less. It seems to be the result of gross incompetence, which is always grounds for firing. There might be other factors at play, but it seems like Jay was "given" responsibility for this system after the first failure, and he completely reneged on that responsibility. If he didn't want that responsibility, he could have made it known at the outset and looked for a better fit of a job if his manager wouldn't help him out. But, so far as everyone in the company was concerned, Jay was responsible for keeping the one-day outage from repeating itself. He did let it happen, and presumably would have continued to let it happen for many more days had the other end of the chain not started asking where those reports were.
Would I have done differently after the first failure? I can say definitely so. I've owned up to bigger blunders that were my own fault, and pushed for process reforms with both more and less at stake. None of us are in his shoes, precisely, but if you've never faced an ethical question in your life you haven't been paying enough attention to what's going on around you or have been exceedingly lucky in life.
All that having been said, on the day of his wedding having to own up to something that was largely his fault (the folks who didn't get their reports for 29 days and didn't raise any alarm bells until they had to sum them up for the monthly bill also likely should have seen something, although without more details there it's hard to say), and cost the company real money, well, that's a much harder question. In my mind, though, that's in the "moral of the story" portion of the tale. Had he been reasonably ethical at the start of this whole endeavor, and/or been competent at his job, he wouldn't have gotten there. I should hope I'd make the right decision there, but it's so far outside the realm of my experience I can't say for certain. Moreso, though, I would hope that I would never get into that particular bear trap in the first place, and the reason I haven't gotten anywhere near it yet is because I make better decisions earlier on.
BTW, this comment box is much better than the old trying-to-be-WYSIWYG monstrosity!
Admin
Your advice is perfectly reasonable for someone working at a company where employees are treated decently and are encouraged to bring their concerns up with the mgmt and to take responsibility for their own foul-ups.
If the salary numbers are correct, chances are the incident took place sometime shortly after The Crash. When bills have to be paid each month and quitting the current job means an avg. of 6 months to find a new one (which pays more than $8/hr, that is), how many people here can honestly say that wouldn't serously consider the same path that Jay took? Not that I'm supporting Jay's actions, but the US is a dog-eat-dog society and not everyone has the means to carry on without getting their hands dirty at one point or another.
The bottom line is, Jay and his employer were obviously a match made in heaven. A company with such a neglectful attitude and hostile working environment deserves employees like Jay who will not hesitate to cover their rear by fudging a month's worth of billing data, if that's what it takes to keep their sole source of income.
Admin
Tell us the whole story, Jay!
Admin
You can't tell me that the person who phoned Jay to fix the problem didn't have some idea that the data was destroyed. The only difference is that he passed the buck to Jay.
When I was younger I had a bosses tell me, "I'm not going to tell you to do that [because it would be illegal], but you know what you need to do." I forget now, what the issue was even, but I remember the quote. These days I have money saved up, and I can find work easily but in those days it was hard. Work can be stressful.
Depending on the data, Jay might have been able to see if he was over charging anyone. From a pratical perspective if a customer complained it would be easy to appologize, make an excuse and give them a refund. Polling is going to be error prone anyway.
Admin
People, we've just been shown the other side of the coin.
Admin
The Real WTF(tm) was that he's getting married :-)
Admin
I can't decide which act was the more stupid: to actually do this work while on leave preparing for his wedding, or to falsify billing information.
When are we programmers going to get some f&*%ing backbone and say, "sorry, I am not available", or simply refuse to take work calls when not at work or on-call.
What is wrong with you guys anyway, did you suck on your mamas tit too long ?
Admin
I'd sorta notice if my bandwidth monitoring platform died. Something about all the pretty graphs vanishing from the projector screens would be rather obvious. Even if Jay's bandwidth monitoring wasn't produced from the same source as the billing data, surely the data from the bandwidth monitoring system could be used to produce more accurate invoices than pure guesswork.
Admin
This was a weeb hosting company. They certainly care about the traffic!
Admin
It looks to me like faking data wasn't Jay's only WTF. As he's clearly responsible for the reports, then why didn't he ever care to find out how exactly they were generated? That guy hasn't been doing his job since day one, is my guess.
Maybe the WTF is how Jay could've survived a company with an apparantly twitchy triggerfinger for so long? ;]
Admin
Why has noone even looked at a box solely responsible for >$1m per month of revenue in the time it took someone to get promoted 3 or 4 times?
The guy deserves the sack at the least and the company doesn't deserve the money if it doesn't even know where it comes from.
Admin
Tick the box on the fornt page next to 'Display Full Article Text'.
Admin
So basically management creates an environment of fear which causes spineless underlings to hide problems. This doesn't excuse said underlings for acts of fraud, but it surely doesn't speak well of management.
Realistically, if people would ignore management threats of getting fired and report these problems then two things can happen. The company goes under due to noone working there any more, or they change their policies. Of course, it is precisely this problem which requires better protection for employees. Not because employees have any inherent right to the job, but becuase incompetent management creates an environment that may cause significant economic damage. That is bad for everyone.
This Jay is a spineless underling and deserves the boot. If he had reported the first time it went wrong, with or without his proposed fix, then maybe the problem would've been fixed.
Oh, any company calling me the day before my wedding better be having the biggest crisis imaginable, and be prepared to pay double overtime with a cherry on top.
Admin
If this is a monthly billing system then nobody would be looking for the data until end of month. If the server crashed at beginning of month then it is simple to not notice for 29 days. And then it is urgent because they want to send the bills out ASAP. You don't ask your telephone company for your bills each day do you? You wait for them to send you the report (bill) at the end of the month. Only then if it didn't appear would you wonder where it was.
Admin
I semi-agree with you, especially about the second instance. But the first instance it sounds like he didn't even know the system existed until someone told him it was broken. I can see how being fired for not being omniscient might inspire one to commit a bit of fraud. The right thing to do is to go to management, and if they want to make the stupid decision to fire you over it, you get to find a place that has less stupid management. But, I can see being a bit weak and committing fraud instead.
The second time though is totally inexcusable on several different levels. It was his problem. He knew it was his problem. It was his fault the system broke in such a way that it was missing data for so long before anybody noticed. It was his responsibility to make management aware of the situation and get the system replaced with something more robust. Fudging the data then is a total evasion of responsibility as well as likely being a fairly major criminal offense.
Admin
That's what everyone told me when I made noises about quitting my increasingly wtf-ish job - and I listened to them, but because the job consumed so much of my energy that I never got around to finding another job I ended up stuck there until I had to get myself signed off with stress-related illness.
The next time I worked in software, it took me a week to find a new job - despite essentially not having done anything for a year. If I'd just quit cold, I probably wouldn't have noticed the gap without pay.
So I'd advise anyone finding that their job is demanding too much from them to resign straightaway, even if it gives them a temporary problem financially - because they'll find a new job much faster if they're not trying to do the old one at the same time.
Admin
"Jay" is unethical and spineless, and if he was one of my employees I would fire him on the spot for a) lying to me and b) committing fraud and c) gross negligence. This is the kind of behavior that leads to the kind of lawsuits that bankrupt companies overnight.
I'm shocked by the happy-go-lucky, "wink-wink-nudge-nudge" tone of this article - it seems that Alex thinks Jay's behavior is amusing, expected, and acceptable. I hope that Alex would feel differently if this was one of HIS employees, and I certainly hope that Alex's customers don't need to expect this type of unethical attitude from him and his company.
Addendum (2007-01-09 09:00): To be clear, I don't THINK that this is indicative of Alex or his company - I don't know him and haven't worked with them before... but if I was a potential client reading this article, I would be taken aback by its tone and have to seriously consider whether or not this is what I could expect. I strongly suggest that Alex should either re-write the article or, more reasonably, DELETE it.
Admin
BWAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHA!!!1!!!
I worked at a chemical company that used FOCUS databases to keep track of MSDS information and when to send it out to customers as required by law. It had a serious bug in it that resulted in the MSDS sheets not being sent out as required by law for almost 10 fucking years. There's a huge fine for each infraction, not to mention the possible injuries to the folks handling the shit.
Each of the divisions used a separate DB to keep track of each of their customers so once the first one went in, they were simply cloned with no testing. I got this dumped on me right before I quit and spotted it right away in testing.
The result? It was handed off to an outsourcing firm to fix. As far as I know it never got fixed since I got a phone call about it 9 months after I quit.
PS: A typical run for 5000 customers would run about 3 hours. Multiply that by 15 or so divisions. I wish I had saved off some of the code, it was the perfect example of how to obfuscate in a very simple 4GL.
Admin
Awesome story :-)
Admin
I whole-heartedly agree, leave all the "wiziwig" crap and the "Niceties" to the non-professionals. I'm sick of that stuff getting in the way...
</not sarcasm>
Admin
This is how things like Enron get started - add some fudge here a little fudge over there - before you know it the boss has a 7 figure pension and Joe Worker is evicted from his single-wide.
Alas there is hope: read about Sarbanes-Oxley.