The Daily WTF: Curious Perversions in Information Technology

2011-01-19 Reply Admin

no name:
Is anyone else visualizing these two as Penn and Teller?
(No disrespect intended - I'm a huge fan of Penn and Teller.)

Actually, I was imagining something more along the lines of Lord Voldemort and Peter Pettigrew.

2011-01-19 Reply Admin

Tapcon:
airdrik:
frits:
The Article:
...no one really knew where the gurus’ cubicles were located. Some believed they were on the 6th floor, a handful were certain it was the 2nd floor, others said the 13th floor, and a few swore they worked out of the Poughkeepsie offices.

Did anyone check Floor 7½?

No, they're on Platform 9 3/4

Which is part of Grey 17 sector I think.

You're all wrong. They work on the 19th floor.

Go find Miss Zarves in HR, she can help you out.

boog · 2011-01-19 Reply Admin

no name:
Is anyone else visualizing these two as Penn and Teller?

Only because the little phantom sidekick never spoke. I wonder what he might have said. Could have been interesting...

The tall man continued in an unemotional tone, “it was not appropriate to have this project be 2008. It is medium risk, and therefore should have been in 2005.”
"Bwak! 2005! Bwaaak!" said the shorter man.

Or maybe not.

2011-01-19 Reply Admin

I have a bit of a side question - why are things anonymized here? Why not just say "We'll use whatever story you submit, so if you want to avoid it coming back change details as necessary". I wouldn't be surprised if people routinely anonymized their own story, followed by Alex doing the same, which results in something incomprehensible.

I mean, sure, Alex change replace any company names with Initech or whatever, but beyond that why should he care? Leave it to the original submitter to take whatever precautions they think are necessary.

-- Note from Alex: ok so a bit off-topic, but to answer your question... it's about telling a story. As they say in Discover Magazine's Vital Signs column, "The cases described are real, but names and certain details have been changed."

It's not so much anonymizing, but conveying a complex scenario (this one took place over many, many months) in an engaging, entertaining, and digestable several-hundred-word article that represents the intersting parts of what actually happened. There are, however, plenty of "raw" stories in the sidebar...

2011-01-19 Reply Admin

Several comments seem to think they're black. The article says "dark-skinned". You can have dark-skinned white people (e.g. Spanish versus Norwegians), dark-skinned Indian people (e.g. Tamil versus other areas), dark-skinned black people, dark-skinned anyone really. I'm quite sure the author is not assigning race here.

dtech · 2011-01-19 Reply Admin

Anonymous:
Awesome story, I really hope it hasn't been embelished too much because this is the exact definition of "worse than failure". If I were David I would never speak of this again - I certainly wouldn't be sharing it on TDWTF. If the story is to be believed, his fake ticket was indirectly responsible for millions of dollars worth of damage and a severe risk to human life.

According to your logic, a jury who won't be intimidated by the mob is responsible for the deaths of his family members...

The engineers were the one and only responsible party for this, they caused a machine to explode, not Dave or anyone else.

2011-01-19 Reply Admin

If David would have pulled that crap on me, I would have blown up a lot more than an MRI machine.

dtech · 2011-01-19 Reply Admin

Mason Wheeler:
I can't believe no one's pointed this out yet, but TRWTF is using a library with no source available. That's one of the things on my "never do under any circumstances" list, precisely because it leads to problems that you aren't able to debug, like in this story.

Often you don't really have a choise. They thought the library was 3rd party developed and 3rd parties aren't too snappy to give away their source code to anyone who asks.

That said, it usually is completely unnecessary to have the source code: as long as everything is documented and works you couldn't care less about the source code. If there is a bug it is usually more effective to just file a report than to try and fix the bug yourself (assuming the lib is indeed complex and you haven't written it yourself)

Mason Wheeler · 2011-01-19 Reply Admin

dtech:
Mason Wheeler:
I can't believe no one's pointed this out yet, but TRWTF is using a library with no source available. That's one of the things on my "never do under any circumstances" list, precisely because it leads to problems that you aren't able to debug, like in this story.
Often you don't really have a choise. They thought the library was 3rd party developed and 3rd parties aren't too snappy to give away their source code to anyone who asks.

They usually are if you're willing to pay for it, IME at least.

That said, it usually is completely unnecessary to have the source code: as long as everything is documented and works you couldn't care less about the source code. If there is a bug it is usually more effective to just file a report than to try and fix the bug yourself (assuming the lib is indeed complex and you haven't written it yourself)

Yeah, that sounds good in theory. In practice, everything is not documented, (not correctly at least,) and if you find a bug in there, you don't have time to wait for howeverlong their development cycle takes, and then maybe have to pay for an upgrade.

Again, never under any circumstances should a competent developer use a third-party library with no source available. It's just begging for trouble.

2011-01-19 Reply Admin

Mason Wheeler:
Again, never under any circumstances should a competent developer use a third-party library with no source available. It's just begging for trouble.

Ah. And all DirectX-developers are incompetent then?

Fascinating. I somehow doubt that you know what you're talking about.

I've got another piece of advice:

Never under any circumstances make broad and sweeping statements. They're usually and inevitably wrong.

2011-01-19 Reply Admin

The two men are obviously avatars of Nyarlathotep.

dpm · 2011-01-19 Reply Admin

Anonymous:
There is no doubt in my mind that David has to accept some responsibility for this scenario, it was his team and his orders they were acting under.

If you read it carefully, you'll see that far from ordering them to go, his stated decision was to go himself --- the duo overrode that decision. So even that flimsy shred of responsibility which you keep trying to slap onto Dave fails to hold.

TGV · 2011-01-19 Reply Admin

MaybeThis:
Could be that one. Two men injured while moving an MRI machine.

Sorry to disappoint you, but when you move an MRI, it's switched off, so there's no software running. And it should have been quenched, because there is no way you're going to be able to move a machine with a >1T magnetic field. So this was just a bug in the design of the hardware (and I don't mean silicon chips).

2011-01-19 Reply Admin

Kensey:
Note to all PMP-certified folk out there: don't sign your e-mails as "Smart Guy, PMP" like you have a doctorate or something. ... Why do they all do that?

I'd guess they do it for the same reason all those people with Cisco and A+ certifications do the exact same thing.

2011-01-19 Reply Admin

I am an engineer and work with power plants.

To me, the real WTF is making a safe operation of a piece of pressurized equipment dependent on some parsing of messages to or from a monitoring system.

Message parsing is probably the most unreliable interface you can have in any system. It seems unbelievable that a safety critical system should be based on this.

It is not many years ago that all safety circuits on boilers had to use hardwired logic (in my part of the world, at least). Today, PLCs can be used for safety circuits, but they are specially certified PLCs which are separated from the remaining part of the control system. The programming of those PLCs have to be approved by the certifying agency which is approving the overall safety of the boiler.

2011-01-19 Reply Admin

2 by 2 .. hands of blue ..

they're coming ..

2011-01-19 Reply Admin

Anon:
I don't care if the story was unrealistic. It was funny and made for a nice change of pace from the usual "look at this awful 50kB single-line SQL request" WTF.

Yes, I see enough of those at work. The record here is 154KB select with subqueries.

DaveK · 2011-01-19 Reply Admin

TGV:
MaybeThis:
Could be that one. Two men injured while moving an MRI machine.
Sorry to disappoint you, but when you move an MRI, it's switched off, so there's no software running. And it should have been quenched, because there is no way you're going to be able to move a machine with a >1T magnetic field. So this was just a bug in the design of the hardware (and I don't mean silicon chips).

Of course it could still be the result of a software bug, despite the machine being switched off at the time. Let me show you:

void onQuenchModeClicked() {
  // TODO: add code to open quench valve
  Sleep(TEN_SECONDS);
  // TODO: add code to close the quench valve
  Alert("Helium quenched - machine can be safely shut down and moved now.");
}

2011-01-19 Reply Admin

I think this reads well as a little tragedy or cautionary tale than a straight true story. (Think of the story of Job.)

Our tragic hero Dave, a good and decent programmer, is tested to breaking point by malevolent forces until he succumbs to his darker instincts and resorts to deception (fakes a support ticket) to try and get rid of the demons testing him. But he inadvertently causes a huge disaster. Dave loses his job (a neat 'punishment' for his error), even causes his team to lose their jobs, and the evil forces are never heard from again.

2011-01-19 Reply Admin

ShatteredArm:
There is absolutely no way this is true. ... There is just too much incompetence in this story for it to be remotely believable.

I've had a half a dozen jobs in as many sectors, and I can tell you that this is, unfortunately, *way* possible. I've got a few dim wits running around here that its amazing they can navigate the interface on their shoes, let alone software.

2011-01-19 Reply Admin

Mason Wheeler:
... never under any circumstances should a competent developer use a third-party library with no source available. It's just begging for trouble.

I'm guessing you've used infragistics controls, haven't you?

chrismcb · 2011-01-19 Reply Admin

This reminds me of my last position. Some work was outsourced. The company had one guy who would work onsite every once in a while.

Every time he checked in, he backed out all my changes. It took me a while to convince everyone that these morons didn't know what they were doing. They kept claiming they were following procedure.

Their procedure was, the dev would come to the office. Sync the code on the local dev machine Download the code, from the dev machine to his laptop. Take the laptop to their office, where presumably others worked on it. Come back to our offices. Sync the code on the dev machine. Copy the code from the laptop back to the dev machine. Check in.

Matt Westwood · 2011-01-19 Reply Admin

no name:
Is anyone else visualizing these two as Penn and Teller?
(No disrespect intended - I'm a huge fan of Penn and Teller.)

I was thinking more Colon and Nobbs.

Kensey · 2011-01-19 Reply Admin

Huzzah!:
Kensey:
Note to all PMP-certified folk out there: don't sign your e-mails as "Smart Guy, PMP" like you have a doctorate or something. ... Why do they all do that?
I'd guess they do it for the same reason all those people with Cisco and A+ certifications do the exact same thing.

And in fairness, I've seen those types too, and they're, universally, complete tools as well. (Except the CCIEs -- that actually means something and you have to practically sweat blood to get it. They get a pass on bragging about it.)

2011-01-19 Reply Admin

Matt Westwood:
no name:
Is anyone else visualizing these two as Penn and Teller?
(No disrespect intended - I'm a huge fan of Penn and Teller.)

I was thinking more Colon and Nobbs.

Jake and Elwood Blues

Destruction follows them everywhere

realmerlyn · 2011-01-19 Reply Admin

no name:
Is anyone else visualizing these two as Penn and Teller?
(No disrespect intended - I'm a huge fan of Penn and Teller.)

That's immediately where I went, too. Glad not to be the only ones!

2011-01-19 Reply Admin

As for the names of the Phantom Duo, they might be called "George" and "Ed". Some in the company referred to them by their initials because that's who they were working for. Must have been out of the home office in Poughkeepsie, and that sounds (in the words of mythbusters) Plausible.

Coyne · 2011-01-19 Reply Admin

Proverb: "The only thing more dangerous than a hardware guy with a code patch is a programmer with a soldering iron."

My guess is that "The Phantoms" were hardware engineers and (with mere reinterpretation of the above) there is nothing more dangerous than a hardware engineer building a software interface.

If they were hardware engineers, they were probably considered irreplaceable. (They were clearly immune to normal employement relations.)

This kind of thing happens all the time. If you consider SQL, for example, the original conception assumed that you could simply read through 1.3TB of data in 13G rows, matching the WHERE clause against each row...and do that for each of 10M transactions/day.

It required software experts using B-tree indexes and clever optimizers to make SQL practical. If the original thinkers had prevailed, there would be no such thing as SQL...because there wouldn't be enough hardware on the planet to search one database.

The real WTF was management not insisting that "The Phantoms" keep to hardware and leave the software to those who know how. But then, as I noted above, they were immune to normal employment relations.

2011-01-19 Reply Admin

Anonymous:
We don't know the full story but I interpreted the article as meaning they had done their debugging in the "lab", come to a satisfactory conclusion (which was wrong, since the ticket was fake to begin with) and then rolled out their change.

FWIW, my reading was that Dave intended for them to go to the lab, but they instead chose to go off to the reporting customer and debug on their equipment instead. (That's what I make of the point that the duo had disappeared by the time Dave was ready to book lab time for them).

2011-01-19 Reply Admin

Coyne:
Proverb: "The only thing more dangerous than a hardware guy with a code patch is a programmer with a soldering iron."
My guess is that "The Phantoms" were hardware engineers and (with mere reinterpretation of the above) there is nothing more dangerous than a hardware engineer building a software interface.

If they were hardware engineers, they were probably considered irreplaceable. (They were clearly immune to normal employement relations.)

This kind of thing happens all the time. If you consider SQL, for example, the original conception assumed that you could simply read through 1.3TB of data in 13G rows, matching the WHERE clause against each row...and do that for each of 10M transactions/day.

It required software experts using B-tree indexes and clever optimizers to make SQL practical. If the original thinkers had prevailed, there would be no such thing as SQL...because there wouldn't be enough hardware on the planet to search one database.

The real WTF was management not insisting that "The Phantoms" keep to hardware and leave the software to those who know how. But then, as I noted above, they were immune to normal employment relations.

Or Chemical Engineers, as the case may be.

Shoruke · 2011-01-19 Reply Admin

Mayor Wilson Boog:
If David would have pulled that crap on me, I would have blown up a lot more than an MRI machine.

That does not reflect well on you...

2011-01-19 Reply Admin

Coyne:
Proverb: "The only thing more dangerous than a hardware guy with a code patch is a programmer with a soldering iron."
My guess is that "The Phantoms" were hardware engineers and (with mere reinterpretation of the above) there is nothing more dangerous than a hardware engineer building a software interface.

If they were hardware engineers, they were probably considered irreplaceable. (They were clearly immune to normal employement relations.)

This kind of thing happens all the time. If you consider SQL, for example, the original conception assumed that you could simply read through 1.3TB of data in 13G rows, matching the WHERE clause against each row...and do that for each of 10M transactions/day.

It required software experts using B-tree indexes and clever optimizers to make SQL practical. If the original thinkers had prevailed, there would be no such thing as SQL...because there wouldn't be enough hardware on the planet to search one database.

The real WTF was management not insisting that "The Phantoms" keep to hardware and leave the software to those who know how. But then, as I noted above, they were immune to normal employment relations.

I think it is quite likely that they were hardware guys, and that they really do know hardware design(They did manage to make working hardware after all) but some hardware designers just can't wrap their head around software. I don't know why that is but its true. The solution is to just assign a software developer to them, who can then learn about the hardware interface, and write the correct driver/message system.

On an unrelated note: They were sent to the lab, which I thought of as a place where there is production quality hardware which is used to development and final test. So how did they manage to blow up a mri scanner in production?

And no I don't think David had any responsibility for the accident.

Example: If one of our software testers get drunk and fill in a bogus/wrong bug repport, and I happen to delete the production database while trying to reproduce it, blaming the testers would just be wrong.

ClaudeSuck.de · 2011-01-19 Reply Admin

And the tall tales told why.

Tall tales? High heels? Irish Girl?

ShatteredArm · 2011-01-19 Reply Admin

boog:
ShatteredArm:
There is absolutely no way this is true. In even the most incompetent company, you don't see a manager agree that a library is unusable, and proceed to let the developers of that library completely take over development in his team. Such a manager would not only be incompetent, but also bipolar. There is just too much incompetence in this story for it to be remotely believable.
So your argument for why this story is fake is that a manager who respects the findings of his developers and had his development team taken over by outside developers who apparently have more pull at the company would have to be "incompetent" and "bipolar"?
Interesting...

I misread it; it was actually the VP who said the library was unusable. Which only strengthens my point; are you suggesting that the two "gurus" had more pull than the technology VP?

2011-01-19 Reply Admin

Harold & Kumar.

MarkJ · 2011-01-19 Reply Admin

A Gould:
Anonymous:
There is no doubt in my mind that David has to accept some responsibility for this scenario, it was his team and his orders they were acting under.

I'd disagree - the article makes it pretty clear that the Mystery Men didn't report to him, and were actively just screwing around doing their own thing. (Obvious Clue Is Obvious: no-one can find their desk?!?)

Yes, the bug is fake. But then the question becomes - shouldn't the Experts have sent the bug back after a week or two marked "unreproducable" (or whatever the tag for "I can't make it break that way" is)? Answer: no, because they like getting to hide off in the Lab screwing around unsupervised. (If no-one can find your employee on a regular basis, odds are it's not because they're so Awesome.)

As for the explosion, since we've already established that the Experts liked to overwrite existing code without check-in or merging, the implication I read was that at some point they "found" their bug, pushed through an update (probably overwriting another dev's code), which caused the screwup down the line.

As for the moral of the story, it's to never let yourself be put in charge of a project, but not in charge of the people involved. It's a sure-fire way to end up holding the bag at the end.

I worked on medical device software for a major player in the industry in the late 1970's, when the Intel 8080 was hot stuff.

No one changed firmware in medical devices you connect to someone's bloodstream without it being tested and released to Document Control. Anyone trying to do so risked having their hands broken...

Another issue is that hospitals keep track of service on their equipment (including calibration). For this story, the Phantom Duo would need to provide paperwork to the hospital certifying that the MRI was working properly before they would put it back in service.

As far as the Phantom Duo themselves is concerned, any company that let people like that run amuck without being accountable to someone is begging (and pleading) for trouble.

2011-01-19 Reply Admin

I thought of the two guys from Central Services.

Coyne · 2011-01-19 Reply Admin

An alternate explanation:
On an unrelated note: They were sent to the lab, which I thought of as a place where there is production quality hardware which is used to development and final test. So how did they manage to blow up a mri scanner in production?

An MRI quench is a failure mode that can potentially happen to any mis-handled MRI anywhere, production or not. (It is an inherent hazard with any superconducting magnet.)

Not knowing for sure, my guess would be that they did something to the software that caused the magnet current to change too quickly, as described here (in section "Power supply").

Once the quench has started, it can't be stopped. (It's like having the wings have come off your airplane in flight: The crash is inevitable.) If I were an operator in that situation I would hope the vents could handle it...while running for my life just in case.

2011-01-20 Reply Admin

dtech:
choise

choice

dtech:
try and fix

try to fix

2011-01-20 Reply Admin

Wow those two gurus did their best to keep the library certified.

Global arrays instead of dynamic allocated stack reduce the chance of a stackoverflow. (You don't want those where things can blow up)
It has been certified for VS 2005 so why use 2008.
Error logging, always good.
No unnecessary changes. Don't fix what is not broken, after all it worked until they updated it.

I somehow don't believe that the software outright ignored errors. After all the only bug mentioned in the story was not even real.

So if the only problem was its usability they could have added an additional wrapping layer and left the working certified code alone.

Good riddance, but the real wtf is wasting money to fix bug free code

2011-01-20 Reply Admin

James:
Matt Westwood:
no name:
Is anyone else visualizing these two as Penn and Teller?
(No disrespect intended - I'm a huge fan of Penn and Teller.)

I was thinking more Colon and Nobbs.

Jake and Elwood Blues

Destruction follows them everywhere

Fagotto and Behemoth. http://en.wikipedia.org/wiki/The_Master_and_Margarita Not spam. Honest. Really.

2011-01-20 Reply Admin

The anonymous bloke obviously has an axe to grind as his comments make little if any sense. I suspect he might even be one of the phantom duo, perhaps the outspoken one.

The disaster clearly resulted from the sheer arrogance of the phantom duo and the clear failure of management to confront them and have them be accountable for their actions.

At the end of the day, the company paid the price for its inability to fire this phantom duo.

2011-01-20 Reply Admin

josefx:
Wow those two gurus did their best to keep the library certified.

Global arrays instead of dynamic allocated stack reduce the chance of a stackoverflow. (You don't want those where things can blow up)

It has been certified for VS 2005 so why use 2008.

Error logging, always good.

No unnecessary changes. Don't fix what is not broken, after all it worked until they updated it.

I somehow don't believe that the software outright ignored errors. After all the only bug mentioned in the story was not even real.

So if the only problem was its usability they could have added an additional wrapping layer and left the working certified code alone.

Good riddance, but the real wtf is wasting money to fix bug free code

Are you going for the "Troll of the month" price? And if so I would like to nominate you.

If you are not going for the "troll of the month" I would like to nominate you to "Most likely poster to give the daily wtf new stuff to post"

2011-01-20 Reply Admin

So there is an obvious correlation - the trouble ticket indicated, directly or indirectly, that there was a problem that needed addressing. This false pretense triggered the whole incident and David was responsible for the false pretense.

Obviously there is a causal relation between someone destroying a piece of equipment and someone asking them to investigate this equipment.

There is a causal relation between the taxi driver getting them from the airport to the hospital and them blowing up the MRI as well. That does not make it the taxi drivers fault, because there is no way for a taxi driver to predict his actions would contribute to an exploding MRI.

If you enable incompetent and reckless engineers to work on dangerous equipment, then you create a dangerous situation. Sure they were investigating a fake bug in this one case, but they would have sooner or later done the same with an actual bug. Their recklessness and their incompetence caused the problem, not that they happened to be there in that one specific instance.

That the problem happened in this instance was accidental. However incompetent engineers who are politically well connected, are bound to cause problems eventually. The cumulative probability that something goes wrong, will be very high. That is, why the companies reputation (quite rightly) was destroyed, and that is the lesson you should learn from this. Don't waste your time finding a way to blame the taxi driver.

frits · 2011-01-20 Reply Admin

josefx:
...bug free code

I knew this was a tall tale.

2011-01-20 Reply Admin

I'm sorrym but I didn't understand what they were doing in a hospital in the first place. The story says

"Not only did The Lab house a fully-functioning MRI machine, but it was two states away. It was usually reserved for hardware-level diagnostics, but occasionally software developers found it useful, especially for really tricky bugs."

Weren't they supposed to go to a company LAB, with a MRI machine just for debugging purposes ?

boog · 2011-01-20 Reply Admin

ShatteredArm:
I misread it; it was actually the VP who said the library was unusable. Which only strengthens my point; are you suggesting that the two "gurus" had more pull than the technology VP?

Actually, I'd question whether the duo even needed more pull than the technology VP. As long as they can influence the VP's decisions better than David or Jared, they have all the authority they need. So... do you think it's unlikely that the duo may have played the political game and earned the favor of the execs prior to the events of this story?

2011-01-20 Reply Admin

Why is this comment featured? The first link is an MRI that exploded when it was moved, nothing in the story suggests that the machine was being moved. The second link didn't explode at all and was "tornado damaged". The third link I have no comment on (can't access YouTube at work) and the last link is a completely unrelated machine (MRI's don't use high-energy radiation of the type that causes risk of overdosing) which also didn't explode.

2011-01-20 Reply Admin

way too much fiction in this story.

An MRI machine exploded once. A story was invented to explain the cause in a way to entertain tech geeks. The end.

TRWTF is that they got approval to use Visual Studio 2008 (which was released only a few months earlier if this story is to be believed) for something as critical as this.

TRRWTF is that creating a fake support ticket worked so well to get them out of the building/state and that they spent over a month diagnosing the non-existent issue

2011-01-20 Reply Admin

josefx:
Wow those two gurus did their best to keep the library certified.

Global arrays instead of dynamic allocated stack reduce the chance of a stackoverflow. (You don't want those where things can blow up)

It has been certified for VS 2005 so why use 2008.

Error logging, always good.

No unnecessary changes. Don't fix what is not broken, after all it worked until they updated it.

I somehow don't believe that the software outright ignored errors. After all the only bug mentioned in the story was not even real.

So if the only problem was its usability they could have added an additional wrapping layer and left the working certified code alone.

Good riddance, but the real wtf is wasting money to fix bug free code

Global arrays because the frumpier member of the duo couldn't figure out how to use method parameters. (Ok, not technically global, but the entire library was one class and they were public fields with a "glb" prefix)
where "certified" is defined as "worked on it in vs2005 for months after everyone (including the offender) agreed to use vs2008"
Where "errors" are actually " 'method X was called' trace messages." They did prove occasionally useful if you could find the text file and the "logging" isn't what actually crashed the library. (A common occurance.)
Where "not broken" is defined as "occasionally worked if the moon was in the right phase"

The Phantom Duo

Leave a comment on “The Phantom Duo”