The Daily WTF: Curious Perversions in Information Technology

2011-01-19 Reply Admin

This - by far - the bestest WTF on this site in years! Thanks for sharing!

2011-01-19 Reply Admin

So, can anyone identify the company based on recorded MRI machine explosions? :-)

2011-01-19 Reply Admin

TroelsL:
This - by far - the bestest WTF on this site in years! Thanks for sharing!

I was going to say exactly the opposite. The only WTF is that management couldn't clearly see what everyone else saw daily. Even after being told about it.

This is the first one that I haven't believed in a long time.

There were far too many mystical elements that had no basis in reality, too.

bikeoid · 2011-01-19 Reply Admin

"Christine, Christine - ignore the software from that cruel Raoul and let me debug that for you"

2011-01-19 Reply Admin

WC:
TroelsL:
This - by far - the bestest WTF on this site in years! Thanks for sharing!

I was going to say exactly the opposite. The only WTF is that management couldn't clearly see what everyone else saw daily. Even after being told about it.

This is the first one that I haven't believed in a long time.

There were far too many mystical elements that had no basis in reality, too.

If you actually struggled to separate the obviously sarcastic "mystical" elements from the actual story, you probably shouldn't be here. And by here, I mean the internet, not this site.

Kensey · 2011-01-19 Reply Admin

"As such, they were as critical to the project as a PMP-certified project manager."

So, completely dispensible then.

Note to all PMP-certified folk out there: don't sign your e-mails as "Smart Guy, PMP" like you have a doctorate or something. Or do, but only if you want to be regarded as a tool by everyone around you with a clue. (Why do they all do that? It's like a virus. One day they're making fun of those people with you, then they get infected with a PMP certification and suddenly it's Serious Business.)

2011-01-19 Reply Admin

Wow, this makes a great campfire story.

2011-01-19 Reply Admin

Awesome story, I really hope it hasn't been embelished too much because this is the exact definition of "worse than failure". If I were David I would never speak of this again - I certainly wouldn't be sharing it on TDWTF. If the story is to be believed, his fake ticket was indirectly responsible for millions of dollars worth of damage and a severe risk to human life. He deliberately sent a team of engineers on a wild goose chase that resulted in unnecessary tampering of a medical diagnostic machine and the complete destruction of that machine, along with his company's reputation.

Most people's idea of a catastrophic failure is accidentally deleting the prod DB and having to restore from yesterday's backup. Let's be honest, that really doesn't hold a candle to blowing up a multi-million dollar device and risking the life of your fellow man in a reckless bid to distract your co-workers. David deserves some sort of a prize, both for his honestly and for the sheer insanity of his reckless abandon.

2011-01-19 Reply Admin

Adam:
So, can anyone identify the company based on recorded MRI machine explosions? :-)

The stories are anonymised and this one in particular warrants it more than most, so don't read too much into the details. I sincerely doubt it was an MRI at all, it could have been anything.

jonnyq · 2011-01-19 Reply Admin

Anonymous:
Awesome story, I really hope it hasn't been embelished too much because this is the exact definition of "worse than failure". If I were David I would never speak of this again - I certainly wouldn't be sharing it on TDWTF. If the story is to be believed, his fake ticket was indirectly responsible for millions of dollars worth of damage and a severe risk to human life. He deliberately sent a team of engineers on a wild goose chase that resulted in unnecessary tampering of a medical diagnostic machine and the complete destruction of that machine, along with his company's reputation.
Most people's idea of a catastrophic failure is accidentally deleting the prod DB and having to restore from yesterday's backup. Let's be honest, that really doesn't hold a candle to blowing up a multi-million dollar device and risking the life of your fellow man in a reckless bid to distract your co-workers. David deserves some sort of a prize, both for his honestly and for the sheer insanity of his reckless abandon.

It doesn't matter if it was a wild goose chase. They were sent to debug something they should have been perfectly capable of debugging. Even if the ticket was fake, they should have been perfectly capable of the debugging without a dangerous explosion. After all, they wrote the original interface, right? The phantom duo should have taken all the responsibility. David's failure was not making this blindingly obvious to management a few different times during the project.

2011-01-19 Reply Admin

Is anyone else visualizing these two as Penn and Teller?

(No disrespect intended - I'm a huge fan of Penn and Teller.)

2011-01-19 Reply Admin

It's not really reckless if there is no apparent correlation between "looking for a non-existent bug" and "blowing up an MRI".

Chasing the bug shouldn't have posed a danger - if a hazardous situation was created, then that happened by those engineers investigating the problem.

2011-01-19 Reply Admin

Anonymous:
Most people's idea of a catastrophic failure is accidentally deleting the prod DB and having to restore from yesterday's backup. Let's be honest, that really doesn't hold a candle to blowing up a multi-million dollar device and risking the life of your fellow man in a reckless bid to distract your co-workers.

Funny that you mention that. We were doing some work involving a measurement system on an airplane. The airplane people wanted our manager to put together a risk assessment, with categories like "minor", "moderate", "major", and "catastrophic".

To the airplane people, "catastrophic" meant that the airplane would crash. To our manager, "catastrophic" meant that all the measurement data would be lost.

I enjoyed listening to the airplane people trying to figure out how an "untimely hard drive failure" could cause the airplane to crash. :)

2011-01-19 Reply Admin

I'm going to agree this is fiction. I figure Stuxnet + Jan 7th Story on MRI have caused an somebodies imagination to run away.

2011-01-19 Reply Admin

http://www.refscan.info/2007/02/15/mri-explosion-in-atlanta/

Could be that one. Two men injured while moving an MRI machine.

2011-01-19 Reply Admin

jonnyq:
Anonymous:
Awesome story, I really hope it hasn't been embelished too much because this is the exact definition of "worse than failure". If I were David I would never speak of this again - I certainly wouldn't be sharing it on TDWTF. If the story is to be believed, his fake ticket was indirectly responsible for millions of dollars worth of damage and a severe risk to human life. He deliberately sent a team of engineers on a wild goose chase that resulted in unnecessary tampering of a medical diagnostic machine and the complete destruction of that machine, along with his company's reputation.
Most people's idea of a catastrophic failure is accidentally deleting the prod DB and having to restore from yesterday's backup. Let's be honest, that really doesn't hold a candle to blowing up a multi-million dollar device and risking the life of your fellow man in a reckless bid to distract your co-workers. David deserves some sort of a prize, both for his honestly and for the sheer insanity of his reckless abandon.

It doesn't matter if it was a wild goose chase. They were sent to debug something they should have been perfectly capable of debugging. Even if the ticket was fake, they should have been perfectly capable of the debugging without a dangerous explosion. After all, they wrote the original interface, right? The phantom duo should have taken all the responsibility. David's failure was not making this blindingly obvious to management a few different times during the project.

There's no doubt that the "Phantom Duo" were the direct cause of the issue and it is their incompetance on display here. But still, David instigated the entire situation - his wholly inappropriate and unprofessional actions set these motions going. This is only interesting because the story appears to have been submitted by David himself and that takes some major balls - if I were him I would be harbouring some serious guilt about my role in all this. I certainly wouldn't be able to consider myself blameless and I hope David learnt an important lesson about inappropriately and recklessly diverting resources to suit his own agenda.

2011-01-19 Reply Admin

markstuddock:
I'm going to agree this is fiction. I figure Stuxnet + Jan 7th Story on MRI have caused an somebodies imagination to run away.

http://www.refscan.info/2007/02/15/mri-explosion-in-atlanta/ http://www.dotmed.com/news/story/15100/ http://www.youtube.com/watch?v=1R7KsfosV-o

Also, http://en.wikipedia.org/wiki/Therac-25

frits · 2011-01-19 Reply Admin

The Article:
...no one really knew where the gurus’ cubicles were located. Some believed they were on the 6th floor, a handful were certain it was the 2nd floor, others said the 13th floor, and a few swore they worked out of the Poughkeepsie offices.

Did anyone check Floor 7½?

2011-01-19 Reply Admin

Wait, so the Duo went to The Lab, which I assume is a testing facility based on the description in the story, but somehow ended up at a real hospital, debugging on live equipment? They must have been real good at disappearing to have kept their jobs for so long.

2011-01-19 Reply Admin

st0815:
It's not really reckless if there is no apparent correlation between "looking for a non-existent bug" and "blowing up an MRI".

The article makes it clear that the "Phantom Duo" actually visited the hospital and adjusted the MRI machine, causing it to subsequently explode. This was based on a trouble ticket that indicated there was a problem with the monitoring system. So there is an obvious correlation - the trouble ticket indicated, directly or indirectly, that there was a problem that needed addressing. This false pretense triggered the whole incident and David was responsible for the false pretense.

That's pretty damn reckless in any industry, but remember also that this was a medical diagnostic machine - so even if it hadn't exploded, any unscheduled down-time for maintenance still puts lives at risk. The "Phantom Duo" may have been the monkeys that screwed up but David was the organ-grinder that dispatched them in the first place. There is no doubt in my mind that David has to accept some responsibility for this scenario, it was his team and his orders they were acting under.

2011-01-19 Reply Admin

Anon:
Wait, so the Duo went to The Lab, which I assume is a testing facility based on the description in the story, but somehow ended up at a real hospital, debugging on live equipment?

^this.

I'm just surprised this story wasn't saved for Halloween... as the kind of tale told to scare young programmers around the campfire...

nonpartisan · 2011-01-19 Reply Admin

Anonymous:
st0815:
It's not really reckless if there is no apparent correlation between "looking for a non-existent bug" and "blowing up an MRI".
The article makes it clear that the "Phantom Duo" actually visited the hospital and adjusted the MRI machine, causing it to subsequently explode. This was based on a trouble ticket that indicated there was a problem with the monitoring system. So there is an obvious correlation - the trouble ticket indicated, directly or indirectly, that there was a problem that needed addressing. This false pretense triggered the whole incident and David was responsible for the false pretense.
That's pretty damn reckless in any industry, but remember also that this was a medical diagnostic machine - so even if it hadn't exploded, any unscheduled down-time for maintenance still puts lives at risk. The "Phantom Duo" may have been the monkeys that screwed up but David was the organ-grinder that dispatched them in the first place. There is no doubt in my mind that David has to accept some responsibility for this scenario, it was his team and his orders they were acting under.

Everyone except for Anon above you keeps forgetting that they should have been debugging this in a lab environment .

2011-01-19 Reply Admin

frits:
The Article:
...no one really knew where the gurus’ cubicles were located. Some believed they were on the 6th floor, a handful were certain it was the 2nd floor, others said the 13th floor, and a few swore they worked out of the Poughkeepsie offices.

Did anyone check Floor 7½?

No, they're on Platform 9 3/4

2011-01-19 Reply Admin

I'm kind of curious if they tried to run the tests on the real MRI that allegedly produced the error on the simulator.

If they didn't see a failure, what did they try to do? Change the MRI to reproduce the failure?

Wild goose chase or not, a few simple tests should have showed there to be no problem.

2011-01-19 Reply Admin

airdrik:
frits:
The Article:
...no one really knew where the gurus’ cubicles were located. Some believed they were on the 6th floor, a handful were certain it was the 2nd floor, others said the 13th floor, and a few swore they worked out of the Poughkeepsie offices.

Did anyone check Floor 7½?

No, they're on Platform 9 3/4

Which is part of Grey 17 sector I think.

2011-01-19 Reply Admin

don't care how made up this is, it's 50 times better than working

2011-01-19 Reply Admin

nonpartisan:
Anonymous:
st0815:
It's not really reckless if there is no apparent correlation between "looking for a non-existent bug" and "blowing up an MRI".
The article makes it clear that the "Phantom Duo" actually visited the hospital and adjusted the MRI machine, causing it to subsequently explode. This was based on a trouble ticket that indicated there was a problem with the monitoring system. So there is an obvious correlation - the trouble ticket indicated, directly or indirectly, that there was a problem that needed addressing. This false pretense triggered the whole incident and David was responsible for the false pretense.
That's pretty damn reckless in any industry, but remember also that this was a medical diagnostic machine - so even if it hadn't exploded, any unscheduled down-time for maintenance still puts lives at risk. The "Phantom Duo" may have been the monkeys that screwed up but David was the organ-grinder that dispatched them in the first place. There is no doubt in my mind that David has to accept some responsibility for this scenario, it was his team and his orders they were acting under.

Everyone except for Anon above you keeps forgetting that they should have been debugging this in a lab environment .

We don't know the full story but I interpreted the article as meaning they had done their debugging in the "lab", come to a satisfactory conclusion (which was wrong, since the ticket was fake to begin with) and then rolled out their change. I don't think they just went straight to the hospital with a screwdriver and a polite smile. But incompentance knows know bounds so you might be right - maybe they just turned up unannounced and started hacking away at vital piece of multi-million dollar medical equipment. Maybe.

2011-01-19 Reply Admin

This was clearly photoshopped.

Captcha: acsi - a Chinese knockoff ascii

2011-01-19 Reply Admin

I was thinking more Master-Blaster from Thunderdome.

No More Methane!

Captcha - erat. Is that an electronic rat?

Zylon · 2011-01-19 Reply Admin

"Even his latest project was following woefully behind,"

Buh?

And add me to the list of people who'd vastly prefer to see the raw "unimproved" version of this story.

2011-01-19 Reply Admin

Actually I was picturing Mr. Croup and Mr. Vandemar from NeverWhere by Neil Gaiman.

steenbergh · 2011-01-19 Reply Admin

Tapcon:
airdrik:
frits:
The Article:
...no one really knew where the gurus’ cubicles were located. Some believed they were on the 6th floor, a handful were certain it was the 2nd floor, others said the 13th floor, and a few swore they worked out of the Poughkeepsie offices.

Did anyone check Floor 7½?

No, they're on Platform 9 3/4

Which is part of Grey 17 sector I think.

So their cubicles aren't in Area 51 then?

2011-01-19 Reply Admin

VogonPoet:
Actually I was picturing Mr. Croup and Mr. Vandemar from NeverWhere by Neil Gaiman.

There's a blast from the past

2011-01-19 Reply Admin

Anonymous:
Adam:
So, can anyone identify the company based on recorded MRI machine explosions? :-)
The stories are anonymised and this one in particular warrants it more than most, so don't read too much into the details. I sincerely doubt it was an MRI at all, it could have been anything.

I guess it was a Nuclear Power Plant in Sovjet Russia...

dkf · 2011-01-19 Reply Admin

VogonPoet:
Actually I was picturing Mr. Croup and Mr. Vandemar from NeverWhere by Neil Gaiman.

Well, I was somewhat thinking of Wint and Kidd from Diamonds are Forever, even if that's somewhat less than accurate from the description.

2011-01-19 Reply Admin

Anonymous:
There is no doubt in my mind that David has to accept some responsibility for this scenario, it was his team and his orders they were acting under.

I'd disagree - the article makes it pretty clear that the Mystery Men didn't report to him, and were actively just screwing around doing their own thing. (Obvious Clue Is Obvious: no-one can find their desk?!?)

Yes, the bug is fake. But then the question becomes - shouldn't the Experts have sent the bug back after a week or two marked "unreproducable" (or whatever the tag for "I can't make it break that way" is)? Answer: no, because they like getting to hide off in the Lab screwing around unsupervised. (If no-one can find your employee on a regular basis, odds are it's not because they're so Awesome.)

As for the explosion, since we've already established that the Experts liked to overwrite existing code without check-in or merging, the implication I read was that at some point they "found" their bug, pushed through an update (probably overwriting another dev's code), which caused the screwup down the line.

As for the moral of the story, it's to never let yourself be put in charge of a project, but not in charge of the people involved. It's a sure-fire way to end up holding the bag at the end.

ShatteredArm · 2011-01-19 Reply Admin

There is absolutely no way this is true. In even the most incompetent company, you don't see a manager agree that a library is unusable, and proceed to let the developers of that library completely take over development in his team. Such a manager would not only be incompetent, but also bipolar. There is just too much incompetence in this story for it to be remotely believable.

2011-01-19 Reply Admin

+1 to trwtf. If you don't like what is posted, what are you doing here ? Even if this story is a fable, it is still useful from a developer's standpoint : Never underestimate the overwhelming power of stupidity.

While this story seems outlandish, I read about a couple of software SNAFU with similar results, most notably about an explosion in a foundry caused by forgotten debugging code. I'm sure you can find other examples that would fit today's story.

frits · 2011-01-19 Reply Admin

dkf:
VogonPoet:
Actually I was picturing Mr. Croup and Mr. Vandemar from NeverWhere by Neil Gaiman.
Well, I was somewhat thinking of Wint and Kidd from Diamonds are Forever, even if that's somewhat less than accurate from the description.

At first I was picturing Jesse Ventura and Alex Trebec from this X-Files episode. Then I read these facts:

The Article:
... a dark-skinned, bald man... ...He shared a dark complexion...

After that, all I could picture was Will Smith and Martin Lawrence from "Bad Boys".

2011-01-19 Reply Admin

VogonPoet:
Actually I was picturing Mr. Croup and Mr. Vandemar from NeverWhere by Neil Gaiman.

My thoughts exactly... Please stop reading my mind.

Anyway, funny story, and I find it plausible.

2011-01-19 Reply Admin

I don't care if the story was unrealistic. It was funny and made for a nice change of pace from the usual "look at this awful 50kB single-line SQL request" WTF.

2011-01-19 Reply Admin

ShatteredArm:
There is just too much incompetence in this story for it to be remotely believable.

You're new to IT arn't you. ;)

2011-01-19 Reply Admin

The story has been heavily anonymised, but is rooted in fact. I no longer work there. Thank God.

@Deprecated · 2011-01-19 Reply Admin

A news report of an MRI explosion at a hospital would cause most people to 'chuckle'?

That is waaay beyond "WTF", more like OMG or Holy $#%#$.

2011-01-19 Reply Admin

Anonymous:
The story has been heavily anonymised, but is rooted in fact. I no longer work there. Thank God.

Some people here say Dave should share in the blame for the loss of some very expensive equipment. Since you were there, do you agree?

Mason Wheeler · 2011-01-19 Reply Admin

I can't believe no one's pointed this out yet, but TRWTF is using a library with no source available. That's one of the things on my "never do under any circumstances" list, precisely because it leads to problems that you aren't able to debug, like in this story.

2011-01-19 Reply Admin

Crash Magnet:
Anonymous:
The story has been heavily anonymised, but is rooted in fact. I no longer work there. Thank God.

Some people here say Dave should share in the blame for the loss of some very expensive equipment. Since you were there, do you agree?

No.

2011-01-19 Reply Admin

Hmm. The story is told as if it were fiction, but the actual story sounds pretty plausible. Normal, even. Which is kind of sad.

2011-01-19 Reply Admin

I've seen it done. On more than one project.

boog · 2011-01-19 Reply Admin

ShatteredArm:
There is absolutely no way this is true. In even the most incompetent company, you don't see a manager agree that a library is unusable, and proceed to let the developers of that library completely take over development in his team. Such a manager would not only be incompetent, but also bipolar. There is just too much incompetence in this story for it to be remotely believable.

So your argument for why this story is fake is that a manager who respects the findings of his developers and had his development team taken over by outside developers who apparently have more pull at the company would have to be "incompetent" and "bipolar"?

Interesting...

The Phantom Duo

Leave a comment on “The Phantom Duo”