• Anonymous (unregistered)

    I think "Classic" is analogous to "Fun Sized".

  • Hot In Here (unregistered)

    So I guess the real WTF is that a civil servant actually left a job after screwing up?

  • (cs)

    So wait,

    it was a 3 day weekend. The guy comes in on the last day of the weekend, notices the A/C units had been running this whole time and then decides to shut them off? Even if it hadn't affected the servers, what the hell did he think shutting off all the A/C for a few hours would do if they had already been running for 2 days anyway...

  • (cs)

    Since when is anyone in government (or management for that matter) ever held accountable for any screwups they inflict on others?

    Oh, btw, this guy now gets to sit at home and collect a pension as compensation for the damage he inflicted - wonderful!

  • Lee K-T (unregistered)

    Civil servant working day and night? WTF!?! They don't even work day around here!

  • MRAB (unregistered) in reply to Kermos
    Kermos:
    So wait,

    it was a 3 day weekend. The guy comes in on the last day of the weekend, notices the A/C units had been running this whole time and then decides to shut them off? Even if it hadn't affected the servers, what the hell did he think shutting off all the A/C for a few hours would do if they had already been running for 2 days anyway...

    Have you heard of the saying "Every little helps"? Of course, sometimes it doesn't...

    Captcha "abigo": abbreviation for "I think I better go"?

  • Blablablaadje (unregistered)

    Don't forget the part where he got a keycard...

  • (cs)

    The real WTF is that they got the key card off the facility manager's desk. Locking out access is designed to prevent this kind of thing from happening.

  • CoderHero (unregistered)

    The real WTF is that they didn't have heat monitoring equipment that would:

    1. Shut down servers if it got too hot
    2. Notify somebody that the room had gotten to hot
    3. Electrically shock anybody who isn't authorized to be in the server room in the first place!
  • Anonymous (unregistered)

    I remember this one. I hope that "early retirement" is a euphemism for being beaten around the head with a stack of overheated computer equipment.

    Can anyone follow up on what actaully happened to this guy?

  • Anon (unregistered) in reply to CoderHero

    Yeah, wtf. I worked in a state server room, and while we may be a little more incompetent that your private company... if the server room ever got to 80, you would have everyone and their mom notified and working to fix the problem. (Of course this has happened several times, and we had to shut down most of the servers until the A/C was fixed, and everyone and their mom sat around and played cards getting overtime until that was finished)

  • Botia (unregistered)

    A true classic WTF. From so many different aspects you can say WTF. Any little thing changed and the disaster would have been avoided but neigh.

  • Anon (unregistered) in reply to MRAB
    MRAB:
    Kermos:
    So wait,

    it was a 3 day weekend. The guy comes in on the last day of the weekend, notices the A/C units had been running this whole time and then decides to shut them off? Even if it hadn't affected the servers, what the hell did he think shutting off all the A/C for a few hours would do if they had already been running for 2 days anyway...

    Have you heard of the saying "Every little helps"? Of course, sometimes it doesn't...

    Captcha "abigo": abbreviation for "I think I better go"?

    I think you a word there...

  • (cs)

    So what procedures, other than shut down and go drinking, are in place at Inedo when the A/C conks out?

  • (cs) in reply to Anonymous
    Anonymous:
    I remember this one. I hope that "early retirement" is a euphemism for being beaten around the head with a stack of overheated computer equipment.

    Can anyone follow up on what actaully happened to this guy?

    This being the government, he's now living for free on retirement paid for by tax payers money. Come on, it's government. What do you expect? Responsibility? Accountability? Please....

  • Adam V (unregistered)

    Wasn't this story fake the first time too?

  • (cs) in reply to Anon
    Anon:
    MRAB:

    Have you heard of the saying "Every little helps"? Of course, sometimes it doesn't...

    Captcha "abigo": abbreviation for "I think I better go"?

    I think you a word there...

    No, the phrase is exactly 'every little helps'. It's also the motto of Tesco's, a supermarket chain in Britain (or the UK. Or England. Whatever).

  • Eco!=Dumb (unregistered)

    He should've been more concerned about the local warming.

  • autoplonk (unregistered) in reply to snoofle
    snoofle:
    Oh, btw, this guy now gets to sit at home and collect a pension as compensation for the damage he inflicted - wonderful!

    Right, because if you screw up, you should have the pension you paid into for years taken away from you, and have to die penniless under a bridge.

  • Walter T. Franklin (unregistered)

    Haha, a few people have already argued about this issue as seen here: http://languagelog.ldc.upenn.edu/nll/?p=749

    I still don't agree... but I guess I am on the wrong side of the ocean?

    Captcha: Appellatio - a vast system of mountains in eastern North America.

  • Fast Eddie (unregistered) in reply to autoplonk
    autoplonk:
    snoofle:
    Oh, btw, this guy now gets to sit at home and collect a pension as compensation for the damage he inflicted - wonderful!

    Right, because if you screw up, you should have the pension you paid into for years taken away from you, and have to die penniless under a bridge.

    I think you mean the pension that I and all the other taxpayers paid into, so yes.

    But I'm not completely heartless. He can start collecting right after the costs for the equipment, time lost, and labor have been repaid.

  • Semper Augustus (unregistered) in reply to Lee K-T
    Lee K-T:
    Civil servant working day and night? WTF!?! They don't even work day around here!

    Sure... You get somebody who is classified in a senior, well paid position, but is just a reporting specialist (i.e. knows how to use Crystal, but has to get somebody else to write the simple SQL statements for him). He wants to prove that he is as busy and overloaded as the people who keep Exchange running, so he farts around all day then complains on how he has to work over the weekend to "bang out these reports".

  • Falcon (unregistered)
    And all computers have cooling fans anyway
    I see - an eco-friendliness lecture from someone who doesn't understand fairly simple physics. No WTF there! :-p
  • (cs) in reply to Fast Eddie
    Fast Eddie:
    autoplonk:
    snoofle:
    Oh, btw, this guy now gets to sit at home and collect a pension as compensation for the damage he inflicted - wonderful!

    Right, because if you screw up, you should have the pension you paid into for years taken away from you, and have to die penniless under a bridge.

    I think you mean the pension that I and all the other taxpayers paid into, so yes.

    But I'm not completely heartless. He can start collecting right after the costs for the equipment, time lost, and labor have been repaid.

    Agreed - this wasn't an accident - it was an intentional act - stupidity doesn't excuse responsibility!
  • (cs)

    So that is what Al Gore did after writing his book...

  • OutlawProgrammer (unregistered) in reply to snoofle
    snoofle:
    Fast Eddie:
    autoplonk:
    snoofle:
    Oh, btw, this guy now gets to sit at home and collect a pension as compensation for the damage he inflicted - wonderful!

    Right, because if you screw up, you should have the pension you paid into for years taken away from you, and have to die penniless under a bridge.

    I think you mean the pension that I and all the other taxpayers paid into, so yes.

    But I'm not completely heartless. He can start collecting right after the costs for the equipment, time lost, and labor have been repaid.

    Agreed - this wasn't an accident - it was an intentional act - stupidity doesn't excuse responsibility!

    It's one thing if the guy broke down the door and burned the tech lab to the ground with a flamethrower. This guy was just an idiot who made a big mistake while trying to do something good. I probably would have fired the guy but it doesn't invalidate everything else that he's done for the organization.

  • Rob (unregistered)

    If this story is true (Ha!) there is a lot more wrong here than someone not understanding the importance of ACs in a server room.

    First, the keycard to the server room was unsecured. The whole point of putting in a fancy lock system is so that people can't just 'get in' to the room. The facility manager screwed up big time.

    Second, nobody configured any sort of alerts for when the room was too hot? The servers weren't set to shut down at a certain point? It was just assumed that the AC units could never, ever, ever, in a million years, not be running as expected? That's a huge screw up.

    The way this should of played out is dumb eco-guy wouldn't be able to get into the room to begin with. If he could get in and shut off the AC units, long before any serious damage was caused, someone would have gotten an e-mail/text message/page and been able to turn the AC units back on. Short of that, the servers should have turned themselves off causes no physical damage - just some downtime when everyone gets back in the office and has to wait for the systems to come back up.

  • (cs) in reply to Fast Eddie
    Fast Eddie:
    autoplonk:
    snoofle:
    Oh, btw, this guy now gets to sit at home and collect a pension as compensation for the damage he inflicted - wonderful!

    Right, because if you screw up, you should have the pension you paid into for years taken away from you, and have to die penniless under a bridge.

    I think you mean the pension that I and all the other taxpayers paid into, so yes.
    Nope, the amount of pension he receives is proportional to the amount he has paid himself.

    His money has been used to pay the pension of the generation before him and logically your money will be used to pay him, but this doesn't mean that you have paid into his pension. He has saved for it himself.

    When you go to the bank, do you check the serial numbers on the bills to make sure you withdraw "your" money and not somebody elses?

  • Scott Evil (unregistered) in reply to CoderHero
    CoderHero:
    The real WTF is that they didn't have heat monitoring equipment that would: 1. Shut down servers if it got too hot 2. Notify somebody that the room had gotten to hot 3. Electrically shock anybody who isn't authorized to be in the server room in the first place!
    1. Sharks with frickin' laser beams attached to their heads!!
  • (cs) in reply to Adriano
    Adriano:
    Anon:
    MRAB:

    Have you heard of the saying "Every little helps"? Of course, sometimes it doesn't...

    Captcha "abigo": abbreviation for "I think I better go"?

    I think you a word there...

    No, the phrase is exactly 'every little helps'. It's also the motto of Tesco's, a supermarket chain in Britain (or the UK. Or England. Whatever).

    NationState Britain = new NationState( { England, Wales, Scotland, Northern Ireland } );

    NationState UK = Britain;

    HTH :)

  • (cs)

    [citation needed]

  • RBoy (unregistered) in reply to Anon
    Anon:
    MRAB:
    Kermos:
    So wait,

    it was a 3 day weekend. The guy comes in on the last day of the weekend, notices the A/C units had been running this whole time and then decides to shut them off? Even if it hadn't affected the servers, what the hell did he think shutting off all the A/C for a few hours would do if they had already been running for 2 days anyway...

    Have you heard of the saying "Every little helps"? Of course, sometimes it doesn't...

    Captcha "abigo": abbreviation for "I think I better go"?

    I think you a word there...

    He intentionally not the whole thing.

  • GSV So Much For Subtlety (unregistered) in reply to GCU Arbitrary

    Not quite...

    Great Britain doesn't include Northern Ireland, but the UK does.

  • illtiz (unregistered) in reply to Rob
    Rob:
    Second, nobody configured any sort of alerts for when the room was too hot? The servers weren't set to shut down at a certain point? It was just assumed that the AC units could never, ever, ever, in a million years, not be running as expected? That's a huge screw up.

    The way this should of played out is dumb eco-guy wouldn't be able to get into the room to begin with. If he could get in and shut off the AC units, long before any serious damage was caused, someone would have gotten an e-mail/text message/page and been able to turn the AC units back on. Short of that, the servers should have turned themselves off causes no physical damage - just some downtime when everyone gets back in the office and has to wait for the systems to come back up.

    Meh. Considered that at my last job. It's not at all trivial, mind you. So if those guys did have the funding and the architectural situation (not a given everywhere) to allow for 2 (two!) A/C redundancies, it is OK for them to not have an automatic shutdown procedure. (Mind you, if that misfires, you will have much unnecessary pain.)

    I will give you that alerts should be generated based on temperature alone (independent of whether A/C was properly shut down or not) and the absence of that is a wtf, I agree to that.

  • Morv (unregistered) in reply to GCU Arbitrary

    Not the case...

    UK = {Scotland, England, Wales, Northern Ireland}

    Britain = The big island that includes most of Scotland, England and Wales, but NOT Northern Ireland.

    Great Britain = {Britain + the surrounding small islands} = {Scotland, England, Wales}

    UK = "The United Kingdom of Great Britain and Northern Ireland". (That's the full official name)

    Sometimes 'british' is used to mean 'from the UK', but this is a colloquialism only.

  • pedant #97334 (unregistered) in reply to GCU Arbitrary
    GCU Arbitrary:
    NationState Britain = new NationState( { England, Wales, Scotland, Northern Ireland } );

    NationState UK = Britain;

    HTH :)

    Wrong. UK is short for "The United Kingdom of Great Britain and Northern Ireland".

    So it's actually:

    Region NorthernIreland = new Region() Region GreatBritain = new Region( { England, Wales, Scotland } ); Region UK = new Region( { GreatBritain, NorthernIreland } );

  • (cs) in reply to Fast Eddie
    Fast Eddie:
    autoplonk:
    snoofle:
    Oh, btw, this guy now gets to sit at home and collect a pension as compensation for the damage he inflicted - wonderful!

    Right, because if you screw up, you should have the pension you paid into for years taken away from you, and have to die penniless under a bridge.

    I think you mean the pension that I and all the other taxpayers paid into, so yes.

    I'm a gov't employee and I and all my colleagues pay into our retirement.

  • (cs) in reply to GCU Arbitrary
    GCU Arbitrary:
    NationState Britain = new NationState( { England, Wales, Scotland, Northern Ireland } );

    NationState UK = Britain;

    (sigh)

    UK = "The United Kingdom of Great Britain and Northern Ireland" Great Britain = "England, Scotland and Wales"

    This is pretty basic information, yet you still managed to screw it up while correcting someone. Score!

  • Hatterson (unregistered) in reply to illtiz

    There's a couple things wrong with this story that make me think it's being misunderstood

    1.) 'Triple' redundant systems being accessed within the same room...presumably that means all running off the same main power line. If you're installing triple redundancy at least one should physically separate. 2.) triple 'redundant' AC systems that are "running the entire time I was here" If the backups are indeed redundant rather than just simply additional they should not be running "the entire time"

    This makes me think they didn't actually have redundant systems. They had 1 system that comprised of three separate AC units. Calling this a triple redundant system is akin to saying my 6-cylinder car engine has 5 redundant cylinders.

    However even with that cleared up there's several WTFs. Very few people escape without blame.

    The employee is stupid for tampering with systems that had nothing to do with him. He's obviously not an IT employee ("big computers" and "have cooling fans anyway" if you needed proof) so he should never even think of touching something that is under the IT departments control without, at the very minimum, running it by someone first

    The building supervisor for leaving a keycard to a very sensitive area just lying on his desk.

    The IT dept for not having, at the very least, noticifaction systems in place for the heat.

    The IT dept again for having 6 domain controllers and 4 exchange servers in the same room without thinking of maybe having a backup in a separate room/geographical area. Keep in mind those 10 servers plus a few sun servers plus a server farm (no idea on the actual size) are just the ones that didn't come up, presumably there were more servers in the room. If you're running an operation of that size you need a backup plan for natural disasters that doesn't involve 'order a new server, wait for it to arrive and then restore the backup'

  • Ernie (unregistered)

    The retirement systems of states are like any other pension funds--the employees pay in.

    The one big difference is that politicians find it convenient to "borrow" against the money, occasionally screwing the employees out of their retirement.

    Since his actions involved extreme stupidity rather than actual malice, I would wager that his supervisor found it easier to "suggest" he retire rather than go through termination procedures.

  • (cs) in reply to OutlawProgrammer
    OutlawProgrammer:
    snoofle:
    Fast Eddie:
    autoplonk:
    snoofle:
    Oh, btw, this guy now gets to sit at home and collect a pension as compensation for the damage he inflicted - wonderful!

    Right, because if you screw up, you should have the pension you paid into for years taken away from you, and have to die penniless under a bridge.

    I think you mean the pension that I and all the other taxpayers paid into, so yes.

    But I'm not completely heartless. He can start collecting right after the costs for the equipment, time lost, and labor have been repaid.

    Agreed - this wasn't an accident - it was an intentional act - stupidity doesn't excuse responsibility!

    It's one thing if the guy broke down the door and burned the tech lab to the ground with a flamethrower. This guy was just an idiot who made a big mistake while trying to do something good. I probably would have fired the guy but it doesn't invalidate everything else that he's done for the organization.

    I agree with your last sentence, but I'm not sure the rest of the analogy is appropriate.

    This guy was obviously not in charge of running the server room, ACs or servers; otherwise he surely would have known why the ACs were running and not to shut them down. He went someplace where he likely had no authority to make changes and shut down equipment when he obviously didn't understand why it was running the first place.

    Even though what he did was a mistake, it was a stupid intentional mistake (granted, access control and temp-alarms could have averted the damage, so there is blame to go around).

    A better analogy might be: if a clerical worker in a hospital goes into your room and notices that the heart-lung machine to which you are connected is running, but doesn't understand that if IT stops, then YOU will stop shortly thereafter, and unplugs it anyway to save electricity, are you saying that though it was an honest mistake, that he should not be charged with murder? In this case, ignorance of the law - or the purpose of the affected equipment - is no excuse.

    In a server room, ignorance of the purpose of the equipment is no excuse when you turn it off and cause damage.

    In other words, if you don't understand why something is doing [whatever], and you shut it off and that subsequently causes damage, then you are responsible and should suffer appropriate consequences.

  • jimicus (unregistered)

    Yes I know there should have been alerting systems in place.

    Yes I know modern servers are quite able to shut themselves down if they get too hot.

    But, like all of these WTFs, there's a lot of detail we don't know. Like how recently all this happened, for one thing.

    And temperature warning systems - particularly in a government department - are the kind of thing that definitely does not get purchased until something happens to demonstrate their necessity.

  • Thank EwE (unregistered)

    Haiku: I work for the state someone pays for my pension and it is not me!

  • Programming Praxis (unregistered) in reply to rudraigh
    rudraigh:
    Fast Eddie:
    autoplonk:
    snoofle:
    Oh, btw, this guy now gets to sit at home and collect a pension as compensation for the damage he inflicted - wonderful!

    Right, because if you screw up, you should have the pension you paid into for years taken away from you, and have to die penniless under a bridge.

    I think you mean the pension that I and all the other taxpayers paid into, so yes.

    I'm a gov't employee and I and all my colleagues pay into our retirement.

    It is defined benefit or defined contribution? The private sector is almost always defined contribution. Most public plans are defined benefit and way underfunded. Guess who makes up the difference?
  • Major Damage (unregistered) in reply to snoofle
    snoofle:
    Oh, btw, this guy now gets to sit at home and collect a pension as compensation for the damage he inflicted - wonderful!

    Maybe, but at least he can't inflict his pain on others.

  • icfantv (unregistered)

    I think this is more a OMGWTFWYT than an WTF. (oh my god, what the f**k were you thinking?)

  • anon (unregistered) in reply to Fast Eddie
    Fast Eddie:
    autoplonk:
    snoofle:
    Oh, btw, this guy now gets to sit at home and collect a pension as compensation for the damage he inflicted - wonderful!

    Right, because if you screw up, you should have the pension you paid into for years taken away from you, and have to die penniless under a bridge.

    I think you mean the pension that I and all the other taxpayers paid into, so yes.

    But I'm not completely heartless. He can start collecting right after the costs for the equipment, time lost, and labor have been repaid.

    Maybe we should just burn him at the stake, would be cheaper. You're like the people that make parents pay millions in insurance costs when their kid jumps under a train: a poor excuse for a human being.
  • Jay (unregistered) in reply to Rob
    Rob:
    If this story is true (Ha!) there is a lot more wrong here than someone not understanding the importance of ACs in a server room.

    First, the keycard to the server room was unsecured. The whole point of putting in a fancy lock system is so that people can't just 'get in' to the room. The facility manager screwed up big time.

    Second, nobody configured any sort of alerts for when the room was too hot? The servers weren't set to shut down at a certain point? It was just assumed that the AC units could never, ever, ever, in a million years, not be running as expected? That's a huge screw up.

    The way this should of played out is dumb eco-guy wouldn't be able to get into the room to begin with. If he could get in and shut off the AC units, long before any serious damage was caused, someone would have gotten an e-mail/text message/page and been able to turn the AC units back on. Short of that, the servers should have turned themselves off causes no physical damage - just some downtime when everyone gets back in the office and has to wait for the systems to come back up.

    Maybe that's why this story is featured on worsethanfailure.com rather than on we-had-a-problem-but-the-backup-kicked-in-and-everything-was-fine.com.

    I'm sure that every day there are thousands of incidents where something goes wrong but people take steps to correct it or have well-thought-out backup systems and so the situation is quickly handled. Nobody bothers to collect such stories because they are boring. "I was driving yesterday and the light turned red, so I put on the brakes but the car didn't stop, I went right through the light and ..." might well make the beginning of an interesting story -- perhaps amusing, perhaps horrifying, depending on how it ends. "... so I put on the brakes and the car stopped smoothly. Then the light turned green and I went on." makes a very boring story. If you know a web site that collects and publishes stories like that ... don't bother to post the URL, because I can't imagine wanting to read them.

  • Rob (unregistered) in reply to Jay
    Jay:
    Rob:
    If this story is true (Ha!) there is a lot more wrong here than someone not understanding the importance of ACs in a server room.

    First, the keycard to the server room was unsecured. The whole point of putting in a fancy lock system is so that people can't just 'get in' to the room. The facility manager screwed up big time.

    Second, nobody configured any sort of alerts for when the room was too hot? The servers weren't set to shut down at a certain point? It was just assumed that the AC units could never, ever, ever, in a million years, not be running as expected? That's a huge screw up.

    The way this should of played out is dumb eco-guy wouldn't be able to get into the room to begin with. If he could get in and shut off the AC units, long before any serious damage was caused, someone would have gotten an e-mail/text message/page and been able to turn the AC units back on. Short of that, the servers should have turned themselves off causes no physical damage - just some downtime when everyone gets back in the office and has to wait for the systems to come back up.

    Maybe that's why this story is featured on worsethanfailure.com rather than on we-had-a-problem-but-the-backup-kicked-in-and-everything-was-fine.com.

    I'm sure that every day there are thousands of incidents where something goes wrong but people take steps to correct it or have well-thought-out backup systems and so the situation is quickly handled. Nobody bothers to collect such stories because they are boring. "I was driving yesterday and the light turned red, so I put on the brakes but the car didn't stop, I went right through the light and ..." might well make the beginning of an interesting story -- perhaps amusing, perhaps horrifying, depending on how it ends. "... so I put on the brakes and the car stopped smoothly. Then the light turned green and I went on." makes a very boring story. If you know a web site that collects and publishes stories like that ... don't bother to post the URL, because I can't imagine wanting to read them.

    I'm not disagreeing with any of that. I'm saying that the blame is inappropriately all dumped on one person who couldn't possibly have caused the problem without the negligence of many others.

  • Sarah Palin (unregistered) in reply to Rob
    Rob:
    Jay:
    Rob:
    If this story is true (Ha!) there is a lot more wrong here than someone not understanding the importance of ACs in a server room.

    First, the keycard to the server room was unsecured. The whole point of putting in a fancy lock system is so that people can't just 'get in' to the room. The facility manager screwed up big time.

    Second, nobody configured any sort of alerts for when the room was too hot? The servers weren't set to shut down at a certain point? It was just assumed that the AC units could never, ever, ever, in a million years, not be running as expected? That's a huge screw up.

    The way this should of played out is dumb eco-guy wouldn't be able to get into the room to begin with. If he could get in and shut off the AC units, long before any serious damage was caused, someone would have gotten an e-mail/text message/page and been able to turn the AC units back on. Short of that, the servers should have turned themselves off causes no physical damage - just some downtime when everyone gets back in the office and has to wait for the systems to come back up.

    Maybe that's why this story is featured on worsethanfailure.com rather than on we-had-a-problem-but-the-backup-kicked-in-and-everything-was-fine.com.

    I'm sure that every day there are thousands of incidents where something goes wrong but people take steps to correct it or have well-thought-out backup systems and so the situation is quickly handled. Nobody bothers to collect such stories because they are boring. "I was driving yesterday and the light turned red, so I put on the brakes but the car didn't stop, I went right through the light and ..." might well make the beginning of an interesting story -- perhaps amusing, perhaps horrifying, depending on how it ends. "... so I put on the brakes and the car stopped smoothly. Then the light turned green and I went on." makes a very boring story. If you know a web site that collects and publishes stories like that ... don't bother to post the URL, because I can't imagine wanting to read them.

    I'm not disagreeing with any of that. I'm saying that the blame is inappropriately all dumped on one person who couldn't possibly have caused the problem without the negligence of many others.

    He was a liberal do-gooder that was sure he knew more than the experts running the data-center. He cost the taxpayers an untold sum in his quest to save the planet. You can't idiot proof the world, they keep making better idiots.

Leave a comment on “Classic WTF: I'm Sure You Can Deal”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article