Comment On The Bank Has Spoken!

How about fixing the cable?

How about just fixing the AC?

How about getting aircon?

mm con air

captcha: NINNNNJA

Wait, so I'm a little confused. The fan bumping into the cable was the cause of the original problem that was causing his code to be blamed for doing "wierd things", right?

Maybe it's just too early for me to reading TDWTF *chugs coffee*

It's probably just my stupidity, but did the issue ever get resolved? Was the problem with his module that the UAT room had a dodgy connection and when it was fixed by moving the fan the problem with the code was fixed, or did you just not get to that part?

I'm awfully confused...

Was this fan just placed there? At the time he started testing the code? Or had it always been there, thus the problem always existed?

Alot of holes in this story... Could be he simply chalked it up to the fan while a real problem is still in there...

The part that I love is that the AC was broken in the first place. A Bank should probably have folks who know what they're doing, and have redundant everything.

Or perhaps they don't need it.

I mean .. it's not like they're dealing with real money in real time or anything like that.

His development environment was not the same server for UAT, so when it came time for UAT and someone turned on the fan in that room, the server where his module was running on would have intermittent network failures causing "weird issues", thats just my interpretation.

Hate to be the spoil sport but, maybe the fan is only magnifying a problem that is still not fixed. The inability to handle an exception when the network connection is down.

database junkie:

The part that I love is that the AC was broken in the first place. A Bank should probably have folks who know what they're doing, and have redundant everything.

Or perhaps they don't need it.

I mean .. it's not like they're dealing with real money in real time or anything like that.

I'm hoping that the AC for the servers was functional (otherwise I can't imagine it functioning at all), and that the broken AC was just for the techs (animals really, they don't need AC).

I think what had happened is that all the old stuff passed UAT, then the air conditioning broke and was replaced with the fan that would hit the network cable.

Seeing as how everything worked and passed UAT, no one noticed anything really wrong with the system, blaming any and all lag on the slowness of computers, "it always does that", etc.

However, when someone tried to put something new on the system, either it relied on the connection more, or people started noticing the problem because they were supposed to be looking for problems.

It's a classic "shoot the messenger" situation.

The application should have been built from the start to gracefully handle temporary disconnections... And I hope it's well-known that delays in network propagation and IPC/thread scheduling are more or less arbitrarily large. Monkeys-in-the-server-room protection comes for free if you simply prepare for "unexpected" transient failures from the start.

Fredric:

Was this fan just placed there? At the time he started testing the code? Or had it always been there, thus the problem always existed?

Alot of holes in this story... Could be he simply chalked it up to the fan while a real problem is still in there...

People people! Do READ the story. The fan was there to replace the currently broken air conditioning. And yes, it caused the "weird issues" cos it broke the network connection every time the fan nudged the cable, thus wonderful things happened when the client was connected to the module and the connection is suddenly broken.

Freaking cheeseheads, try to read the whole story before submitting.

That so sounds like something that would happen at our company or one of our customers...ahhhh

I read this as a part being cutted out of the 'Firewall' movie....(wich I just happen to watched yesterday).

Captcha: wigwam (dunno, I think that was another story)

Was this what the server room looked like?

Heh... a typical server room. :D You have to be extra careful when moving around, otherwise you'll get "wired". :P

CAPTCHA: craaazy... indeed... :D

Some code has checks to see if the network connection is up but the "network connection goes down for 3 seconds every 10 seconds" case is rarely checked.

doug:

The inability to handle an exception when the network connection is down.

Correct!

Sorry, but a network-application that can't handle network errors... Hummm... *chchchch*

Well, I know, checking return codes or catching Exceptions sounds like wheenies - but I'd like this :-)

Last but not least:
There exist servers, it's in a bank... so I assume (or hope) they have good Network-guys, ... Why didn't Network-monitoring-software recognise the problem of a network-Connection going up-n-down 5 Times a second ??

jo42:

Was this what the server room looked like?

How the hell did you get into my room?!?!?!?! I'm getting a restraint order from the magistrates! damn you!

"Let's call him 'Steve.'"

It's a pretty name.

a wise old-timer once told me: don't ever work for a bank

(we'll call him "Steve")

I think we should call him Gorak

This reminded me of a story in my early days of programming IBM mainframes in the 1970s. This was still back in the days of punch cards, when some data was stored in punch cards residing in a tub bin.

A programmer tried for months to isolate a problem with the accounting module, which would occasionally leave the corporate books out of balance by just a few dollars each month. One night he was working late, investigating the problem and pouring over the COBOL source code. As he studied the code, the cleaning lady was just finishing with her sweeping in the room. She reached over into the card tub bin, took a card at random, and swept the little pile of dirt onto the card. Then she threw the card and dirt into the trash can.

Problem solved there, too...

Fortunately, he was able to add a FanBumpedTheCableException() class to the code, which could be caught and make the thread sleep for the period of time that the fan was in contact with the cable.

Is there something wrong that I actually thought this would be what happened?

Gedoon:

People people! Do READ the story. The fan was there to replace the currently broken air conditioning. And yes, it caused the "weird issues" cos it broke the network connection every time the fan nudged the cable, thus wonderful things happened when the client was connected to the module and the connection is suddenly broken.

Freaking cheeseheads, try to read the whole story before submitting.

Pretty sure I did, and it was a terrible story at that. Fan was there to replace a broken AC, but wasn't turned on? Issues still being caused when it's not on? Hmm.. Fan comes on, say spinning at 3200rpm, and "Steve" can actually see the network going up and down? The bank actually uses broken ethernet cables? And why the hell is UAT being performed in a restricted access server room, one with no AC at that? Is there any sort of professionalism left in this world?

SomeCoder:

Wait, so I'm a little confused. The fan bumping into the cable was the cause of the original problem that was causing his code to be blamed for doing "wierd things", right?

Maybe it's just too early for me to reading TDWTF *chugs coffee*

I had a problem like this back when 10base2 ethernet was the thing. A customer brought his electrician to install his new 10base2 ethernet and when we brought our new PC to install network was not working. After to and fro and back and forth and fingerpointing without end we had it working. Only one problem remained: when they left the windows terminal emulations using telnet over TCP/IP open then the connections would reset after some time and they would have to re-login.

We searched and could not find any problem - over weeks.

The last time my networking/cabling colleague and me (systems guy) went there our boss told us: "You fix it today or the customer will go to court." Fair enough, considering: the company staff complement was cut to the bone and a virtual madhouse.

At that point we had about fifteen similar customer installations with both 10base2 and 10baset ethernet running and no problems like that at all. That indicated a site/customer-specific problem.

So I was sitting in front of one client PC and waited for things to happen. Sure it did: after half and hour connection reset. I relogin. 10 minutes later - the same. 5 minutes later - again. I could not avoid to look out of the window and finally the dime dropped: every time the connection was reset a heavy truck passed the property.

Now the customer was situated in the fishery harbour in Bremerhaven in Germany with a lot of heavy (up to 32 metric tons) truck traffic running directly in front of the property on account of said handling most fresh fish being brought ashore for Germany. The road was old and every time such a heavy truck would pass the property there was a (more than slight) vibration experienced in the office where I happend to be at. Still I had no clue what to do but fortunately my colleague had been around block on this issue: I told him and he went to check the 75 ohms termination resistors - sure enough: he found a loose contact with one of them.

We showed the owner, fixed it and left.

Man weeks were wasted on this - confound it.

People don't seem to be reading this right.

There's no suggestion that his module, or the system as a whole couldn't cope with network interruptions. Maybe it could, maybe it couldn't - it matters little to the story.

The gist of the story is that at the same time that they introduced his code they must have introduced the fan.

So during UAT, the network periodically dropped - something didn't happen previously and this was reported to the devs only as 'weird things'.

The fan caused the problem, but the bank blamed the code.

It's not rocket science.

Been there...:

This reminded me of a story in my early days of programming IBM mainframes in the 1970s. This was still back in the days of punch cards, when some data was stored in punch cards residing in a tub bin.

A programmer tried for months to isolate a problem with the accounting module, which would occasionally leave the corporate books out of balance by just a few dollars each month. One night he was working late, investigating the problem and pouring over the COBOL source code. As he studied the code, the cleaning lady was just finishing with her sweeping in the room. She reached over into the card tub bin, took a card at random, and swept the little pile of dirt onto the card. Then she threw the card and dirt into the trash can.

Problem solved there, too...

LOL

This should be on thedailyWTF home page.

jo42:

Was this what the server room looked like?

I like the head - is that the head of the previous sysadmin manager ?

olsner:

The application should have been built from the start to gracefully handle temporary disconnections... And I hope it's well-known that delays in network propagation and IPC/thread scheduling are more or less arbitrarily large. Monkeys-in-the-server-room protection comes for free if you simply prepare for "unexpected" transient failures from the start.

That's all very well, but if it is a browser based application, the browser is going to return a server error when the server gets disconnected from the network, since the connection between the browser and server has been broken. No matter how gracefully the application code handles network interruptions, the browser is going to barf.

captcha: pinball

lackluster:

Pretty sure I did, and it was a terrible story at that. Fan was there to replace a broken AC, but wasn't turned on? Issues still being caused when it's not on? Hmm.. Fan comes on, say spinning at 3200rpm, and "Steve" can actually see the network going up and down?

How many fans do you know that OSCILLATE at 3200rpm? And surely that would 3200 opm then? I think you're confusing the rotation of the fan's blades with the (usually adjustable) oscillation of the fan itself to cover a wider area.

It does seem odd though that UAT would take place in an overheated, ultra-secure server room.

A lot of people are complaining why couldn't the application handle network errors... I don't see anywhere in the story that it couldn't... just that it acted wierd... maybe wierd was the application delivering network down warnings, or the operating system delivering network down warnings... after all, the old version didn't do that...

I've more then once seen a new version of an app blamed for things completely unrelated... new version rolls out, 1-3 days later there's a complaint that the screen is flickering now that the new version is deployed... you walk down to see what's going on, and there's a fan beside the monitor, and it flickers every time the magnet gets near the monitor... They don't think about the fan they plopped down beside the computer, they only think of the app as changing, so it MUST be responsible for any oddness that happens.

The real WTF is the idiot who wrote this story and ended it in the middle.

Oscillating, so it travels back and forth. Maybe once a minute the blade housing moves the cable. Everyone has seen a loose rj45 connector on a cable, it happens.

Nothing to do with how fast the blades were moving.

StupidGuy:

doug:

The inability to handle an exception when the network connection is down.

Last but not least:
There exist servers, it's in a bank... so I assume (or hope) they have good Network-guys, ... Why didn't Network-monitoring-software recognise the problem of a network-Connection going up-n-down 5 Times a second ??

Right, so they aren't going to pay for working AC but they are going to shell out for some obscure software that no one in senior management will understand what it does. As a network admin, I wish it were that easy. We've been requesting network monitoring software for so long that entire netmon software companies have come and gone while our request sits.

Also, I must say that usually when a user says "something weird" is happening, I assume (from experience) that it means "I didn't pay attention to what I was clicking on and have no idea the damage I've done". This is one of the few cases I have seen where there was ACTUALLY something weird going on.

Captcha: Pirates...ggaaarrrhhh

Gareth:

People don't seem to be reading this right.

There's no suggestion that his module, or the system as a whole couldn't cope with network interruptions. Maybe it could, maybe it couldn't - it matters little to the story.

The gist of the story is that at the same time that they introduced his code they must have introduced the fan.

So during UAT, the network periodically dropped - something didn't happen previously and this was reported to the devs only as 'weird things'.

The fan caused the problem, but the bank blamed the code.

It's not rocket science.

while(1) { cout << "bears "; }

lmodllmodl:

A lot of people are complaining why couldn't the application handle network errors... I don't see anywhere in the story that it couldn't... just that it acted wierd... maybe wierd was the application delivering network down warnings, or the operating system delivering network down warnings... after all, the old version didn't do that...

I've more then once seen a new version of an app blamed for things completely unrelated... new version rolls out, 1-3 days later there's a complaint that the screen is flickering now that the new version is deployed... you walk down to see what's going on, and there's a fan beside the monitor, and it flickers every time the magnet gets near the monitor... They don't think about the fan they plopped down beside the computer, they only think of the app as changing, so it MUST be responsible for any oddness that happens.

This reply is absolutely spot-on. I've seen the same kind of situation many, many times.

The folks immediately decrying some assumed inability to cope with network problems in the application have more in common with the bank higher-ups in the story than they do with the developer.


CAPTCHA: yummy

I agree

Iceman:

That's all very well, but if it is a browser based application, the browser is going to return a server error when the server gets disconnected from the network, since the connection between the browser and server has been broken. No matter how gracefully the application code handles network interruptions, the browser is going to barf.

That's only true if you use full page refresh. In our application [at the bank of course! :)], there are mainly ajax-style connections to the server, which will yield only a "please try again" warning if the request fails.

The real wtf (and I say that not only in spite of, but because of how many of you feel about the phrase) is that:

The official description was that the module caused "weird things" to happen.

My response would be: I checked it out, weird things are not happening, problem solved. And then proceed to do nothing until a better bug description was forthcoming or I was fired.

anonymous guy:

My response would be: I checked it out, weird things are not happening, problem solved. And then proceed to do nothing until a better bug description was forthcoming or I was fired.

I've tried that several times. Usually QA goes to management who tells me to play nicely.

jo42:

Was this what the server room looked like?

The cut head is a nice touch

I work in a company that has to deal with banks on a very regular basis. I'm moderately sure that the hiring process for the programmer/tech/development people at a bank involves putting 50 people in a room, and hiring the one that drools on himself the most.

I guess this guy's modules were the only ones on that server that access any network resources. That, or all the other modules silently swept their exceptions under the rug.

StupidGuy:

Sorry, but a network-application that can't handle network errors... Hummm... *chchchch*

I don't recall reading anything in the article that said that it didn't. But what user wouldn't report 'weird things' happening when a connection cyclicly goes up and down -- whether app/data integrity is preserved or not?

Sounds like a case of

try
{
doSomething();
}
catch(Exception)
{ /* swallow */}

Been there...:

A programmer tried for months to isolate a problem with the accounting module, which would occasionally leave the corporate books out of balance by just a few dollars each month. One night he was working late, investigating the problem and pouring over the COBOL source code. As he studied the code, the cleaning lady was just finishing with her sweeping in the room. She reached over into the card tub bin, took a card at random, and swept the little pile of dirt onto the card. Then she threw the card and dirt into the trash can.

There does seem to be an entire subclass of computer WTFs based around clueless maintenance and construction workers. My favorite-- computer scientist working on garbage-collection algorithms has all his notes on the subject in a box labeled "GARBAGE". Yeah, do I even need to finish this one?

QA Blog, 2007-02-15

I have finally made a break through on writing a regression test for QA issue 25078: "Fan disrupts network connection." Unfortunately, they will not let me deploy the robot that I built to unconnect network cables on demand.

I may have to come up with a software solution instead...

Captha: waffles but the f's were cut off so it looked like "wattles"
(wattles! shiver me timbers)

shambo:

anonymous guy:

My response would be: I checked it out, weird things are not happening, problem solved. And then proceed to do nothing until a better bug description was forthcoming or I was fired.

I've tried that several times. Usually QA goes to management who tells me to play nicely.

If you're still working there, then apparently you didn't do that! Just kidding, so I'm being a little harsh. First I'd explain why that wasn't a real bug description. And that QA were the ones not "playing nice". If they expect you to fix a problem based on such a nebulous non-description, then I don't want to work there. Time to start looking for another job. So ok, I wouldn't just "wait" to get fired. But you get the idea.

anonymous guy:

shambo:

anonymous guy:

My response would be: I checked it out, weird things are not happening, problem solved. And then proceed to do nothing until a better bug description was forthcoming or I was fired.

I've tried that several times. Usually QA goes to management who tells me to play nicely.

If you're still working there, then apparently you didn't do that! Just kidding, so I'm being a little harsh. First I'd explain why that wasn't a real bug description. And that QA were the ones not "playing nice". If they expect you to fix a problem based on such a nebulous non-description, then I don't want to work there. Time to start looking for another job. So ok, I wouldn't just "wait" to get fired. But you get the idea.

:) I don't work there anymore, I found something better. It was a case of bullying QA and pushover management. Most of the bugs submitted where either non-reproducible (or not documented on how they were reproduced) or where changes. It wasn't much fun to be a developer there.