• xix (unregistered)

    We used to keep the doors to the server room open (We being the help desk) to enjoy the AC, and we used to get in crap just for that (the room was in no danger of warming up to dangerous levels).

    People should pass little intelligence tests before having access to keys to such places though.

  • hmi (unregistered)

    well, that's a laugh

  • SomeCoder (unregistered)

    I sure hope by "early retirement" you mean he was severely beaten before he was fired for being a complete dumb ass.

    I can't imagine how upset I'd be if I were in that IT department.

  • Ares (unregistered)

    Al Gore?

    CAPTCHA - onomatopoeia - almost made me not want to comment

  • burninator (unregistered)

    Who wants to bet that this just resulted in an admin page and someone turning the A/C back on. Alex probably hyped up the failure for extra drama

  • stace (unregistered) in reply to xix

    Wow. Is this supposed to be an example of irony?

  • Otter (unregistered)
    Comment held for moderation.
  • whicker (unregistered)

    Are newer rack servers able to protect themselves better than the ones in this room?

    Several hundred 50-cent temperature sensors and power cutoff switches embedded in various things (power supplies, boards, disk drives) have got to cost less than a tertiary a/c.

    No, I know you people just purchase whatever is available, and have little if any say in the parts that go into the equipment. But by now, things have got to be perfected to the point where you can add a server farm to your shopping cart and have it arrive next day.

  • Ben4jammin (unregistered)
    With all the power that all those big computers in that room use Gee...I wonder what is causing all the heat that is making these AC units run constantly I doubt it is really eco-friendly to run those big units at the same time. I'm a liberal! I'm smucking fart! And all computers have cooling fans anyway, so why put the A/C for the building in that room? I need to point this out for all the dumb power-hungry conservatives in our IT dept In the future, let's try to be a little more conscientious of our energy usage! I just saved $4 dollars' worth of electricity and it only cost us about 200 grand in equipment!

    No, dill-hole, you didn't make Mother Earth smile...

  • Atrophy (cs) in reply to whicker
    whicker:
    I know you people just purchase whatever is available, and have little if any say in the parts that go into the equipment. But by now, things have got to be perfected to the point where you can add a server farm to your shopping cart and have it arrive next day.

    Man I wish I lived in your world. Around these parts, your new domain controller is the the HP Pavilion from 4 years ago that the boss's kid doesn't want anymore.

  • dennis moore (unregistered)

    interesting.. that showed up as a posting on the livejournal techsupport community right after memorial day weekend from someone who claimed to work at the office where it happened. the post no longer exists.

  • dbnoob (unregistered)

    come and witness the daily wtf! as seen before on livejournal's community "techsupport", weeks ago.

  • Pap (cs)

    http://forums.worsethanfailure.com/forums/thread/122464.aspx

    Dupe!!

    CAPTCHA: False Associationism -- how appropriate! :)

  • Otterdam (cs) in reply to Pap
    Pap:

    *Dope!!

    It's not a dupe when it appeared on the sidebar. Several front page stories started there.

  • akatherder (cs) in reply to whicker
    whicker:
    No, I know you people just purchase whatever is available, and have little if any say in the parts that go into the equipment. But by now, things have got to be perfected to the point where you can add a server farm to your shopping cart and have it arrive next day.

    It also depends what vendor. If you order a Sun server with 4 GB of memory and add 4GB more, they put it in another cardboard box and ship it along with the server.

  • Erzengel (cs)

    The Facility Manager should be fired for leaving his keycard somewhere where an unauthorized person could access it and so gain access to several hundred thousand dollars worth of equipment, which he proceeded to "accidently" destroy. If he couldn't access the server room, he wouldn't have been able to shut down the AC. There is also the backup--have the computers send your cell phone a text message (or beep you in earlier times) when they start to fail. ("Oh my god, I can't talk to ExServ123! ExServ123 just died! Someone help, I just lost ExServ345! Gack... byuuu....") I've heard people doing this for all sorts of technical problems.

  • Mike (unregistered)

    Obvious fake.

  • Pat (unregistered)

    It's bad enough that he did it, but to send an email to send a self important message like that to IT is what makes this guy a real asshole.

  • James (unregistered)

    HAHAHAHAH!!! Whooo.... hippies are dumb.

    Wait, it's a fake?

    ...

    "Fake but accurate"?

  • Erik (unregistered)

    The idea that that many critical servers would not have some sort of capability to signal someone if they failed is ludicrous. Is the IT team totally dependent on user complaints to find out when a server goes down during working hours?

  • Jeff S (cs) in reply to xix

    Boys and Girls -- just simply ignore the "frist!" posts. They are eventually deleted ... Don't complain, don't comment, just ignore them. They go away, they really do, I promise.

  • Number6 (unregistered) in reply to Mike
    Mike:
    Obvious fake.

    It has to be a farce.

  • Bob Jones (unregistered) in reply to whicker

    Just wait until those high-quality 50-cent temperature sensors start shutting down your domain controller and it takes you three or four hours to figure out that the 50 cent temperature sensor is bad and has kept you from doing any real work because some tight ass couldn't fathom spending money on a tertiary A/C.

  • anne (unregistered)

    A liberal did a dumb thing. Therefore, all liberals are dumb.

    Nice logic there. I hope you're not programming anything important.

  • KattMan (cs) in reply to anne
    anne:
    A liberal did a dumb thing. Therefore, all liberals are dumb.

    Nice logic there. I hope you're not programming anything important.

    That programmer made a flawed assumption, therefore all programmer assumptions are flawed.

    This one actually makes sense, because if the spec doesn't spell it out and the programmer has to make a decision on something himself, it is always the wrong way for someone.

  • T$ (unregistered) in reply to Erik
    Erik:
    The idea that that many critical servers would not have some sort of capability to signal someone if they failed is ludicrous. Is the IT team totally dependent on user complaints to find out when a server goes down during working hours?

    This is a really good point, at the least there should be some sort of automated email system for the temperature of the room, I couldn't imagine the cost being all that high.

    The bigger issue as I see it is either: a) ----- should have access to the room. If this is the case, why did you hire someone that would make that big of a mistake so easily to do a job where they'd need that kind of access? b) ----- should NOT have access to the room. In this case, why is there no lock on the server room door to keep such people out?

  • snoofle (cs) in reply to Jeff S

    Not so sure this is fake. Yes, we've all heard the urban legends, but I've gotten more than one page from my systems telling me that this or that alarm has triggered (along with some relevant critical numbers). Invariably, it's because some idiot did what s/he wasn't supposed to do.

    Many years ago, I had built a simple web-entry system (the department had us use pure html, cold fusion and a sybase DB). Like most folks, I created a script to create and initialize the DB. In the script, I followed the typical: drop, then re-create pattern. When everything was working, I changed the script to print out a big "Do not run this as it will destroy the DB" message (at the top) and then exit, before all the actual DB stuff. For extra safety, I made the script read-only.

    A few months later, I was put on another project, and some new guy was brought in to maintain the system. One day, my old boss comes over to me and asks me to help them figure out why the system seems to have lost much of its data. I go to enter one ticket and notice the sequence numbers have been reset to the starting value. I ask the obvious question, and the guy tells me that he saw the script was read only, so he made it executable. He saw the message, and commented it (and the 'exit') out. Why? Because he wanted to see what it did. headdesk

  • Ben4jammin (unregistered) in reply to anne
    anne:
    A liberal did a dumb thing. Therefore, all liberals are dumb.

    Nice logic there. I hope you're not programming anything important.

    I assume this was in reference to my post, since I used the word "liberal" (you know there is a quote button, right?)

    I in no way inferred anything about "all" liberals. I merely attempted to inject humor into a thought process that was obviously incomplete in its logic. True, I made an assumption that "eco-friendly" was more likely to be used by a liberal than a conservative...call it poetic license and chill out. Luckily I don't program anything, I am a network admin. So don't worry about my programming skills. I am smart enough to know that if AC units are constantly running, you might want to figure out what is causing the heat before shutting them off.

  • ParkinT (cs) in reply to xix
    xix:

    People should pass little intelligence tests before having access to keys to such places though.

    Remember, this is a state agency. These are Government employees. It would be a violation of (some sort of) civil rights to test intelligence; as would be the case with competence. {And, probably, cause a Union hubub too}

  • pfarrell (unregistered) in reply to Ben4jammin
    Ben4jammin:
    >With all the power that all those big computers in that >room use

    Gee...I wonder what is causing all the heat that is making these AC units run constantly

    I doubt it is really eco-friendly to run those big units at the same time.

    I'm a liberal! I'm smucking fart!

    And all computers have cooling fans anyway, so why put the A/C for the building in that room?

    I need to point this out for all the dumb power-hungry conservatives in our IT dept

    In the future, let's try to be a little more conscientious of our energy usage!

    I just saved $4 dollars' worth of electricity and it only cost us about 200 grand in equipment!

    No, dill-hole, you didn't make Mother Earth smile...

    Lighten up Francis

  • snoofle (cs) in reply to pfarrell
    pfarrell:
    Ben4jammin:
    >With all the power that all those big computers in that >room use

    Gee...I wonder what is causing all the heat that is making these AC units run constantly

    I doubt it is really eco-friendly to run those big units at the same time.

    I'm a liberal! I'm smucking fart!

    And all computers have cooling fans anyway, so why put the A/C for the building in that room?

    I need to point this out for all the dumb power-hungry conservatives in our IT dept

    In the future, let's try to be a little more conscientious of our energy usage!

    I just saved $4 dollars' worth of electricity and it only cost us about 200 grand in equipment!

    No, dill-hole, you didn't make Mother Earth smile...

    Lighten up Francis

    Great reference - thanks for the laugh!

  • MattS (unregistered)

    My former boss did something similar in a client's datacenter. Unfortunately he should have known better and didn't even have the eco-excuse, he just got cold while working in there and forgot to turn them on again before he left.

  • Otter (unregistered) in reply to snoofle
    Not so sure this is fake.
    Not fake as in something like this has actually happened? Probably.

    Not fake as in the text of the email being real? Unlikely.

    Not fake as in the presentation here having the slightest relation to the way this really played out? No, someone simply submitted a much-repeated urban legend.

  • Berislav Lopac (unregistered)

    And scientists still wonder where all that global warming is coming from?

    Captcha: bathe (Gee, I've heard of intelligent software, but this is too much. And I don't even have a Webcam.)

  • redbeard (cs) in reply to Erik

    Could be. Considering I'm a part time admin with no pager and no cell phone and no way to tap into the phone system anyway, I don't have anything like this.

    There are four other people actively involved with our servers, two of whom do it full time. A few weeks back a water leak blew up the main power distribution panel feeding the server room. That caused the generator to kick on. Everything at that point was fine, but no one was notified. Then, at about 2 a.m., a hose blew off the generator, which promptly shut down. Again, no notifications went out. One of the full time IT people got a call as he got out of the shower at 7:30, "Hey, did you know everything is down?"

    So, yes, the IT team could be totally dependant on user complaints. That sort of thing costs money. And if it's a little too much, it doesn't get added.

  • slapout (unregistered)

    This reminds me of when I worked at a state agency (different state) and they were trying to figure out how to save money. They decided to cut power to the building for the weekend. But they forgot to tell the IT department about it. So when everyone came in Monday morning, we couldn't get any work done until about noon when they got the network back up.

  • Erzengel (cs) in reply to T$
    T$:
    b) ----- should NOT have access to the room. In this case, why is there no lock on the server room door to keep such people out?
    Read the letter again: it WAS locked, but ----- went and stole the key from [the facility manager’s] desk. This is why I say [the facility manager] should be fired, leaving a key where an unauthorized person can access it is worse than turning off 3 AC's.
  • Moldylocks (unregistered)

    Reminds me of when they got the thermostats reversed for the server room & the programmers room. As the servers (AS400's) warmed up it got colder in our room, which kicked on the heat in the server room... It was around 120-130 degrees when someone finally went in (because a PC-based print server died). The 400s never even hiccuped.

  • Bart B (unregistered) in reply to redbeard
    redbeard:
    Could be. Considering I'm a part time admin with no pager and no cell phone and no way to tap into the phone system anyway, I don't have anything like this.

    There are four other people actively involved with our servers, two of whom do it full time. A few weeks back a water leak blew up the main power distribution panel feeding the server room. That caused the generator to kick on. Everything at that point was fine, but no one was notified. Then, at about 2 a.m., a hose blew off the generator, which promptly shut down. Again, no notifications went out. One of the full time IT people got a call as he got out of the shower at 7:30, "Hey, did you know everything is down?"

    So, yes, the IT team could be totally dependant on user complaints. That sort of thing costs money. And if it's a little too much, it doesn't get added.

    I really wonder when people don't stop and think "someone must have come up with a solution for this?". You know how 'much' server monitoring costs? NOTHING! It's called Free and Open Source Software, in this case, the excellent Nagios.

  • Bart B (unregistered) in reply to redbeard
    redbeard:
    Could be. Considering I'm a part time admin with no pager and no cell phone and no way to tap into the phone system anyway, I don't have anything like this.

    There are four other people actively involved with our servers, two of whom do it full time. A few weeks back a water leak blew up the main power distribution panel feeding the server room. That caused the generator to kick on. Everything at that point was fine, but no one was notified. Then, at about 2 a.m., a hose blew off the generator, which promptly shut down. Again, no notifications went out. One of the full time IT people got a call as he got out of the shower at 7:30, "Hey, did you know everything is down?"

    So, yes, the IT team could be totally dependant on user complaints. That sort of thing costs money. And if it's a little too much, it doesn't get added.

    I really wonder when people don't stop and think "someone must have come up with a solution for this?". You know how 'much' server monitoring costs? NOTHING! It's called Free and Open Source Software, in this case, the excellent Nagios.

  • Nick (unregistered) in reply to Bart B

    Nagios may be free, but it is far from excellent.

  • Doug (unregistered)

    Whoever turned off the A/C should have just offset the energy usage by planting a few Eucalyptus trees in Uganda.

  • quad (unregistered) in reply to Doug

    or by saving us all the trouble and just killing themselves thereby reducing all further things they could have possibly done to hurt the earth.

    save a tree, kill a hippie

  • nwbrown (cs) in reply to Ares
    Ares:
    Al Gore?

    CAPTCHA - onomatopoeia - almost made me not want to comment

    When will people learn, just because someone claims to have invented the Internet* doesn't mean they are qualified to run your IT infrastructure.

    On a more serious note, he claims he took the key card from the facility manager's desk, WTF was it doing there? Shouldn't something like that be locked up so people like Mr Gore can't get in?

    *yes, I know the actual quote was that he was "took the initiative in creating the Internet" so you can stop whining

  • Bobp0303 (unregistered) in reply to Erzengel

    When I worked at Eaton in the early 80s, upper management all smoked. At least one of them regularly smoked in the computer room. As a junior programmer who didn't happen to smoke, I asked him if that was such a good idea because the (300 megabyte washing machine) disk drives ran at such tight tolerances. There was a poster at the time that showed that disk heads 'flew' lower than a human hair :) :) His reply was mostly non-verbal -- he dropped his cigarette on the raised flooring next to the closest disk drive and commented disdainfully that "Disk drives have HEPA filters." Something less than a month later we had a disk drive failure compounded by a less than knowledgeable operator taking the trashed disk and trying it in most of the other drives before he was stopped. -- After the massive cleanup, a no smoking rule was posted. I wasn't senior (or ballsy) enough to ask the management individual how he felt about having contributed to the mess. The money didn't count -- we were working on a DOD security project.

  • AdT (unregistered)

    I vote for a fake. I don't doubt that one day an idiot would get up and do something totally stupid like turning off the AC in a server room. Well, actually, I believe that every day, an idiot gets up and does somthing that stupid.

    But the e-mail - I don't believe it. Not only does this guy steal a key to get unauthorized access to the server room, in order to switch off the AC without authorization, but then he boasts about it on the company mailing list? And chastizes his coworkers for leaving the ACs running?

    And the most suspicious thing: He destroys $200k worth of equipment and they merely lay him off instead of also suing the sh** out of him?

    Then someone pointed out that similar but not pin-pointable versions of this story have been circulating for a while - this is very common for urban legends. I think this is the Republican version of the "Bush holds children's book upside down" or "Bush forgets to take caps off binoculars" legends, fabricated to make fun of $POLITICAL_OPPONENT, in this case "liberal tree huggers", obviously.

    If the story were actually true, the Real WTF would be that in a server room stuffed with hundreds of thousands of dollars there is no temperature sensor that can trigger an automatic shutdown and service personnel alerts.

  • Franz Kafka (unregistered) in reply to AdT
    AdT:
    I vote for a fake. I don't doubt that one day an idiot would get up and do something totally stupid like turning off the AC in a server room. Well, actually, I believe that every day, an idiot gets up and does somthing that stupid.

    But the e-mail - I don't believe it. Not only does this guy steal a key to get unauthorized access to the server room, in order to switch off the AC without authorization, but then he boasts about it on the company mailing list? And chastizes his coworkers for leaving the ACs running?

    And the most suspicious thing: He destroys $200k worth of equipment and they merely lay him off instead of also suing the sh** out of him?

    Then someone pointed out that similar but not pin-pointable versions of this story have been circulating for a while - this is very common for urban legends. I think this is the Republican version of the "Bush holds children's book upside down" or "Bush forgets to take caps off binoculars" legends, fabricated to make fun of $POLITICAL_OPPONENT, in this case "liberal tree huggers", obviously.

    If the story were actually true, the Real WTF would be that in a server room stuffed with hundreds of thousands of dollars there is no temperature sensor that can trigger an automatic shutdown and service personnel alerts.

    You must be fresh out of college or something. I've met enough self righeous idiots to totally believe this, and enough penny pinching managers to believe they'd leave out the monitoring and alarms.

  • Mr Steve (unregistered) in reply to ParkinT
    ParkinT:
    xix:

    People should pass little intelligence tests before having access to keys to such places though.

    Remember, this is a state agency. These are Government employees. It would be a violation of (some sort of) civil rights to test intelligence; as would be the case with competence. {And, probably, cause a Union hubub too}

    LIBERAL government employees must be mega stupid then?

    In my super liberal country (New Zealand) government workers actually get paid more then the private sector. Sure governemnt is big and dumb, but the people who work there are mostly pretty on to it.

  • Bubba (unregistered)

    Well, at a former job, we had a machine that would send messages to a another machine that was offsite. If the offsite machine did not hear a message every 2 minutes or so, we'd get paged.

    Of course, that cost money. The way I've seen governments work (thru colleges) is that if you can't directly attribute a cost of something to a project, it doesn't happen. So, it wouldn't surprise me if there were no resources for monitoring, as there would be no budget for it.

  • James (unregistered)

    That is eco terrorism! OMG.....

Leave a comment on “I'm Sure You Can Deal”

Log In or post as a guest

Replying to comment #:

« Return to Article