- Feature Articles
- CodeSOD
- Error'd
-
Forums
-
Other Articles
- Random Article
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
Admin
Admin
First up, I can't imagine that someone would be fired just because a legacy system they didn't impliment, on ancient hardware, fell over for a day.
Sh-t happens. Everyone knows that. He should have reported it in email to his manager, along with recommendations on how it could be avoided in future (replaced the hardware with something more modern, rewriting of the app, whatever). Doing so would have established a paper trail and handballed responsibility up the chain.
Yep, his managers might have decided to average out previous data or have negotiated with the customer or whatever. But they get paid to exercise that responsibility and fundamentally, that's what they're there for.
Jay's problem is that he acted dishonestly. An error's forgivable, especially when reported for further action to resolve it. Committing fraud for personal gain (keeping his employment secure) is a serious no-no and likely both a sacking and criminal offence.
Admin
Meh. This is the kind of story that would be really funny as a Dilbert strip, but isn't quite as funny when it actually happens in the real world.
Admin
The only reason I can think of for Jay not telling his management chain would be that he was directly responsible; during his "mining" he himself deleted the log by accident, or something similar.
I'm a little bit shocked he didn't bite the bullet the first time around.
Admin
You actually thought he would make all the .aspx pages him self?
Admin
Shirley, you can't be serious!
Admin
Can you do fancy routing in asp.NET, or does Alex really have separate pages for each story?
Its quite easy to implement url rewriting in .net...
Admin
It wouldn't happen in his company ... because he doesn't have a trigger-happy management to begin with!!! Basically, this is a story about a WTF within a WTF ... showing that bad management can lead to this.
Jay was not even "in charge" of this. He wasn't even responsible for the server, no one actually told him to "check it out". Though the curse of "you touch it, now you maintain it" might apply...
I've seen dudes being made to do far worse things at their jobs (the kind that would not only land you in jail, but in nationwide news scandals), just because their boss told them to do it or join the "Unemployment Club".
Admin
Mama always told me not to lie and cheat. Mama was wrong. I have found from experience that the liars and cheaters always get ahead, and the honest get fired. Mostly true for sales people.
Admin
The real WTF is that he didn't use fractal interpolation to fudge the data. Subtracting 6-11% every day??? WHAT!? Does that mean the third monday he fudged was 35% below the previous quarter's monday average?
Admin
what else? file not found?
Admin
I earned the everlasting ire of a department head at my last job for reporting to a superior the facts that a) we had been over paying something by ever increasing amounts, and b) I had been asked to slowly fudge it into being fixed rather than fix it all at once so as to not cause a major upheaval when those payments dropped to the correct amounts.
If I had it to do over, I think life would have been much easier if I would have just gone with the fudging.
Captcha: tango
Admin
But anyone can be a responsibility-evading schmuck, it doesn't take WTF management (or any management) to do that. (Not to imply Alex ever would be.)
Admin
" Nowhere does the article say he might have been fired for reporting that the server was at risk "
I don't think that it needs to be explicitly written. You can understand that he is nervous from the tone of the writing. For instance, what if management were to ask him how he knows that the system needs to be rewritten? Just cause something is old, does not mean it needs to be rewritten. But if something fails, then maybe it does. Now, mgmt will say, hmmm, so how long did this fail for? I can understand his nervousness.
Admin
You can imagine this happening to varying degrees of severity every minute in some company somewhere. Three things I see.
What Jay did was dishonest and should probably be fired (put on notice or probation or whatever is done at his company at the very least).
There is obviously a cultural flaw in the company where someone cannot report a problem or error without fear of reprisal. I briefly worked in such company where we would meet everyday at 6:00pm to discuss issues and work through solutions that came up with operating the website but it usually turned into a CEO rant on how stupid we were. This naturally discouraged any useful open and frank conversation.
Jay's manager should also be severely reprimanded. Delegation with out any kind of employee monitoring to whether something is getting done correctly or not is just plain abdication of responsibility. Jay's boss should have a process in place where the bandwith monitoring is verified every day.
Admin
This happens in Spain all the time. The electric meters are inside the house, so you have to let the man in to read it. So some people pretend to not be home when he arrives. Then, since he doesn't have the real number, they guess at the usage. This is what you get billed
Admin
Jay was a Unix admin who was too lazy to write a relatively simple shell script to ensure (or at least warn about the lack of) system health on a single box? WTF!
Admin
Or use ISAPI_Rewrite, I'm shocked no one has mentioned this...or did I miss it?
Admin
Assigning blame in this store is much more easy than some of you make it sound. There are three distinct possibilities of responsibility:
a) Jay never knew about this server, as he wasn't told about it and he wasn't responsible for 'everything'. In that case, there is absolutely no reason why he should fear termination in even the most trigger-happy surroundings. A simple "The data is gone because management failed to make the server someones responsibility after the initial tech guy quit" would have all he had to say.
b) Jay never knew about this server, as he wasn't told about it but he should have known because everything in his department was his responsibility. If that was the case, it was Jays fault from the get-go, because he neglected to inquire what that little machine in the corner was doing. If you are responsible for "all dem servas", then thats what you are responsible for. Not feeling like finding out what half of them did is no excuse.
c) Jay was told about the server but forgot or was simply too lazy to deal with it. This, I would say, is by far the most likely possibility, as Jay already demonstrated almost criminal neglect when he found the error, did not fix it, planned on making sure it didn't happen again - and then DIDN'T. He knew what was going on, he knew what to do each day to prevent it, and he didn't. And don't tell me you forget stuff this serious for a month. He was simply too lazy.
In all cases, the fault can clearly be assigned to someone, and since Jay didn't report it right away, it was clearly not option a).
As for Alex, I'm at a loss. Is he not legally required to report a crime of this magnitude ? Is not having this post up itself practically an outcry to get sued by some DA ?
Admin
At one job I got into trouble over and over from "accidentally" adding my boss's boss to the CC list when there was really bad news. My boss would yell at me and I'd appologize and claim it was a typo and promise never to do it again.
It made life hard though. Sometimes my boss would get a reprimand and sometimes he wouldn't. Either way, it made life hard.
Sometimes I did fudge things, though. So I can't fault Jay. It sounds like everyone got a discount. None of his team got into trouble. Hopefully he destroyed all the evidence and came up with good cover stories for every eventuality.
Admin
If your meters are like the ones in the US that would only work until they managed to catch you at home and got the real number at which point you'd be billed/credited for the difference. My meter's outside but the power company only reads it roughly 2 out of every three months. Really annoying since I'm using roughly double the power of the previous tennant and even after 18mo the estimates are still esentially based on her usage and the lowball numbers mess with my monthly budgets.
Admin
The right way to fake this would be to use an autoregressive linear predictive error filter. This is a variety of filter often used for 'noise reduction': to reduce or cancel out noise, you have to be able to predict a seemingly random process from its history. Basically you train the error coefficients of the filter from the data you have and use it to estimate what values the process will produce. But from the sounds of it, nobody was doing any serious data analysis on the bandwidth stats, so nobody would have noticed if you had just made a random trend yourself.
Admin
Oh we wouldnt want that now would we :P
Admin
Yes.
ASP.NET (only in 2.0 - though I believe) allows something called "URL Mapping". Often also refered to as "Friendly URL's"
Try a Google search for "asp.net url mapping", you'll get lots of relevant results.
Admin
I'm one of the few who thinks Jay didn't really have a choice here.
Saying anything about the true situation = fired, no change in the result unless he makes such a big splash that the whole company is destroyed by it. (And how is that going to look on your resume??)
Furthermore, management either wanted him to fudge it or didn't care about the true situation.
Thus, fudge it and start job hunting.
Admin
"didn't have a choice" has probably been tied to more acts of evil than any other phrase in history. It is how weak people justify doing things they know is wrong. There are few if any situations where people can't make a choice. It is what makes us human - the ability to balance our needs against other forces.
Your own need for money is so great that you are justified in potentially over-charging your companies customers to ensure it?
Let's roleplay:
Your December electricity bill is very high. You are having some trouble making ends meet, so you are very frugal in January, cutting your use to 1/4 of December. However when your bill comes for January it's cost and usage is very near the December amounts - clearly an error. Do you call to complain? Would you demand a refund even if it meant someones deception being uncovered and their firing?
.
Admin
Surprising to see such an apparently large company with no failover plan what-so-ever. Shocking!
Admin
Jeez, guys, get a grip....Jay should be promoted to CFO...they do this all year long, making up numbers and, maybe, screwing the customers...but that what they're there for...paying the bills. Jay should have at least gotten a top-end Porsche out of this and a 7 figure Beverly Hills home....with maybe a Rodeo Drive charge card for the understanding new bride......
Admin
Judging by his description of the management, they wouldn't have taken responsibility for this. They would have fired Jay, possibly sued him, and left the system for someone else to find.
The only way your "duty of tech employees" works is if the management is willing to accept criticism AND take responsibility.
Admin
I'm old fashioned. I'd just have a script listening at the top-level of the site, that would dispatch to functions based on the values of the '/'-separated components of the PATH_INFO environment variable. Simple as a pimple.
Catcha: foxtrot
The best of the old Peter Gabriel era Genesis albums...showing my age a bit, huh?
Admin
I don't see how they could have sued him over it but they certainly would have fired him and then done something like what he did anyway.
My take on situations such as this is that I only think one has responsibility for a situation where one can reasonably change the outcome. I don't see how Jay could have with destroying his future job potential--and he shouldn't be required to take a hit like that in response to a problem committed by others.
Admin
Admin
I would tell the boss about the problem the first day it happened, and propose to implement a better solution that operates on one of the main parts of the infrastructure (code for a linux router would be the lowest-level superleet way to go). This way, if that went down, it didn't matter becuase the whole system would be down anyway.
captcha: random
Admin
Admin
The better way is to use nginx - the server designed to raise multi-user Apache/IIS perfomance by lowering their requirements in RAM :-)
Admin
Ahhh, i know how he must have felt like. Before the days of flatrates i worked for an ISP too, and we had a realy crappy traffic counter programm, that crashed everytime the temp file exceeded 2GB.
We had a nice script that rolled the file to another location where it was analysed, and another temp file was created.
I was watching the computer every day, and of cause, nothing happened, until i was away for 3 or 4 days. Nearly EVERY TIME i was away, that fucking script failed. And i had to "recreate" the logs.
I realy feel with this guy.
Admin
Most of the comments regarding the unethical nature of this are complete idiocy. This is a management WTF if anything. They don't want to vigilantly evaluate critical systems and ask questions, with putting a lowly Unix admin in charge of a mission critical application, ignoring it for months, and then having the ego to fire people for problems that they themselves don't have the follow-through to handle. I've worked in an environment like this.
If you want to fire someone for this error, first fire the CIO for not covering a mission critical system, then the sysadmin manager for not having an inventory of critical systems, backups, and redundant coverage for them as well as making them visible all the way up the chain. Then have the successor instruct the sysadmin on better escalation procedures while he's correcting the issue.
If I was this guy, I probably would have done the exact same thing (except probably not let it go that long). I would have falsified the reports, then when I got back, attempted to fix them while I circulated my resume to work at a place that doesn't suck quite so badly. I'd discuss my reason for leaving as seeking an environment where things where a little more professionally organized, and that the lack of redundancy and coverage in previous position made me nervous. With any luck I'd still get good references out of previous management for my new job as they would not yet have discovered the falsifications.
Admin
URL rewriting in .NET is easy. There's built-in functionality in .NET 2.0, and previously you just use a HttpHandler.
Admin
<quote user=Pete>First up, I can't imagine that someone would be fired just because a legacy system they didn't impliment, on ancient hardware, fell over for a day.</quote> Ohh, I wish I lived in your world. I once had the joy of working for a company with dotted lines in the heirchy. A project - A - was requested & approved by the dotted VP. After 3 months of work on A, the project was cancelled by the VP of my dept - with repeated emails back & forth reguarding the ramifications. When the dotted VP found out his project was not ready at due date, he went to the CEO. When the CEO came to the project manager with demands as to why it wasn't ready, out came the E-mails from the VP cancelling the project. When the smoke cleared, the project manager was gone and the VP's both had more dotted lines to other depts.
Admin
Wow... Jay is WAY out on a limb here.
One thing I haven't seen anyone consider so far is that Jay probably WILL be fired for something or other on down the road. And when that happens, he will likely believe that it is because the management is quick to fire people for "minor" transgressions, or for being the messenger. In fact, I wouldn't be surprised if this has happened to him before, which may color his perception of his current situation.
I have to wonder if the management is really as quick to fire as is described in the article, or is Jay's sense of what is and is not a fireable offense severely warped.
Admin
Fair enough. The argument can be made that they should all go. But so should Jay. The difference is, Jay is likely also legally liable. So, his higher ups get fired, he gets fired and sued. Falsifying billing information even when directly ordered to do so can get you in very hot water legally. Doing it on your own, with no direction from higher up, is just incredibly stupid.
Admin
The one thing you don't quite get is that although you agree the argument can be made that they should all go, how realistic do you think the picture is that would really happen? More likely Jay goes bye-bye, and the chain up from him protects their own ass and manage to squarely use their membership in the "good ole' boys club" to filter the info to avoid their own accountability and blame Jay.
Jay's legal liability is pretty far down the chain of risk management at this point. Even a lowly sysadmin can manufacture some data and cover his tracks. If his artifacts are nowhere to be seen on the falsified reports, I still maintain his best course of action is falsify, baby, and do the same as his predecessor. Abscond for a promotion elsewhere. Even if it is eventually discovered, do you think all the management chain is going to admit it and expose their ineptitude? They would have some legal exposure in the due diligence category.
This company is so completely clueless that this mission critical system was written by someone who was promoted up the chain to a top level position, never fixed it, and now is a CTO somewhere. That's the real WTF.
Fix, smile, obtain CIO's job recommendation elsewhere, and never look back upon that house of cards collapsing.
Admin
Just because you haven't seen environments like where Jay worked doesn't mean they don't exist. Management is clueless. Their perception is that the person reporting the problem is the cause of the problem and simply trying to avoid the blame.
Admin
Argh! He was a sysadmin, and didn't even think to automate checking of the report thingy?
Presumably, the bloody thing generated files on a daily basis. Simply check to see if those files are created, or if it is a file that just grows - check the last modification date, and have it mail the results to your email account! Better yet -- have that same script check the process status table - if the app is dead the script can restart it!
He would have caught it, and not had to fudge anything...
Geez! These things are programable, you know.
Admin
Yes there is a much better way, in asp.net -- it's been a while but you can just create a class that implements IHttpHandler or IHttpModule.
See http://www.devx.com/dotnet/Article/6962/0/page/1
Admin
So I was supposed to be doing a valve replacement on a guy, right? But I and most of my team were out playing golf. So I get this call from the hospital that they need me in the OR pronto. Time to think fast. Knowing that 85% of all valve replacement patients die within the same timeframe as they would without the replacement, I use my crackberry to update the patient's records remotely and tell the nurse, "We did that last week! Check the records!" Nobody noticed, and my game went perfectly!
Admin
The sad irony is HAD HE REPORTED THE PROBLEM HE WOULD HAVE BEEN LIABLE FOR THE DAMAGES AND POSSIBLE JAIL TIME!!!
Sorry, to the people standing up for "ethics"... Ethics have no place in the budget of commercialism. Most of the time the good deed becomes the handmaiden of the scapegoat.
At best he could have done is resign without reason the first time the event occurred... However I'm dead certain he resigned to "live somewhere quiet" a few weeks after the wedding.
He must of brought this up at least once or twice in a two quarter period. That was more then enough for covering HIS moral and legal grounds... However employers have the budget to hire lawyers at any whim even when just losing money due to stupidity cough SCO cough
Admin
I can't say I'd have done the right thing in this case. I don't know if I'm a strong enough person to do it. But I damn well know what the right thing is, and Jay didn't do it.
Admin
The word FRAUD comes to mind. and Fraud worth 1.45M would land Jay and everyone else in the federal pen.
Holy crap.