• Zygo (unregistered) in reply to Jerome
    Jerome:
    akatherder:
    whicker:
    No, I know you people just purchase whatever is available, and have little if any say in the parts that go into the equipment. But by now, things have got to be perfected to the point where you can add a server farm to your shopping cart and have it arrive next day.

    It also depends what vendor. If you order a Sun server with 4 GB of memory and add 4GB more, they put it in another cardboard box and ship it along with the server.

    OMG, they ship it along with the server? That's service for you!

    If you'd ordered it from HP, they would have sent the memory along separately from the server, and it wouldn't arrive until three months later. When you rang them up to complain, they would have told you they don't sell that sort of memory in the first place.

    In my experience.

    In my experience, you'd receive (by FedEx) a 12x18x3" box full of anti-shock foam, inside which you would find a large anti-static bag, inside which you would find an HP-branded folder, inside which you would have a two-page letter thanking you for purchasing HP memory, and informing you that the memory can be delivered to the address of your choice if you phone the 1-800 number and speak to a sales representative (be sure to quote the memory part numbers precisely) and thank you for buying HP.

  • George Orwell (unregistered) in reply to savar
    savar:
    You could at least write a script which runs on each machine and pings all the other machines every minute...sends an email/text message/whatever when a server disappears.

    Put at least one server on a UPS (they're not that expensive!) so that if the power supply fails all of them simultaneously, there is still one alive to send the SOS. If you can't afford a UPS, then use a laptop.

    Help me! When power goes off my laptop is unable to send out the emails. Simply logs "Can't find route to host"
  • (cs) in reply to burninator
    burninator:
    Who wants to bet that this just resulted in an admin page and someone turning the A/C back on. Alex probably hyped up the failure for extra drama

    I understand that "Alex made up 99% of the story" makes for an awesome new running gag (Almost as awesome as "captcha") but he already stated in a previous comment that he keeps the anonymisation to a sane minimum. So could we maybe stop repeating this in every single story?

  • (cs) in reply to KattMan
    KattMan:
    anne:
    A liberal did a dumb thing. Therefore, all liberals are dumb.

    Nice logic there. I hope you're not programming anything important.

    That programmer made a flawed assumption, therefore all programmer assumptions are flawed.

    [image]

  • VAR (unregistered) in reply to Jerome
    Jerome:
    akatherder:
    whicker:
    No, I know you people just purchase whatever is available, and have little if any say in the parts that go into the equipment. But by now, things have got to be perfected to the point where you can add a server farm to your shopping cart and have it arrive next day.

    It also depends what vendor. If you order a Sun server with 4 GB of memory and add 4GB more, they put it in another cardboard box and ship it along with the server.

    OMG, they ship it along with the server? That's service for you!

    If you'd ordered it from HP, they would have sent the memory along separately from the server, and it wouldn't arrive until three months later. When you rang them up to complain, they would have told you they don't sell that sort of memory in the first place.

    In my experience.

    That is why you never by directly from the manufacturer. They are good at building things, not customer service. That is where a Value Added Reseller comes in.

  • Voiceofthelost (unregistered)

    Hey Author....

    Way to rip off my post from LiveJournal and make it sound like it's yours.

    Whoever submitted this can kiss my fat hairy ass.

    Fuckers.

  • (cs) in reply to unklegwar
    unklegwar:
    From this point forward, all postings will be suspect for BS in my eyes.
    Including this one of yours?
  • Fred the Fourth (unregistered)

    Things may be better now, but back in the 80's I was personally involved in two similar incidents. In one case, one night Facilities shut down the evaporative coolers that were remotely supplying cold water to the computer room, and the ambient temp reached about 120F in the room. There were no automated call systems in those days, but a late-working programmer noticed (WTF?) the heat before any damage occurred. In the other, a well-meaning security guard shut off the main power on a large test oven, which was actually keeping the prototype system inside (relatively) cool. In that case, Murphy's law kicked in and the system's power supply fried itself before the (faulty) overtemp sensors tripped. $60,000 for a new PSU.

  • Zygo (unregistered) in reply to Robert
    Robert:
    By the way, a room full of servers like that with 3 running A/C doesn't go only to 109F. And even if it was that low a temperature in the room, a computer can run at 109F just fine.

    Google/wikipedia "Newton's law of cooling", or for that matter, "the second law of thermodynamics." (yes, yes, I know, some Creationists don't believe in the second law of thermodynamics, but they're not exactly running server rooms are they? ;-)

    Cooling efficiency goes down as ambient temperature goes up. The internal temperature of critical components might be near the failure point with a machine's cooling system already operating at full capacity (or at whatever capacity is left after three years worth' of hair and dust is sucked through the cooling system) and an ambient temperature of 25.6 degC (78 degF for the Americans). Add a few degrees to the outside world and the machine simply falls over.

    Power supplies are the worst offenders here. A power supply basically has two ways to fail: it can disconnect the line voltage from the motherboard completely, or it can connect the line voltage to the motherboard very directly. Power supplies are designed to preferentially fail in the former direction, but I've seen some that fail in the latter, magic-smoke-releasing direction, especially if they got too hot some time in the past.

    Not that I would particularly trust the hardware after an event like this even if it actually survived--if it got hot enough to fail, there's a good chance that some kind of lasting damage occurred as well, so I wouldn't be in a hurry to throw it back into production.

  • AdT (unregistered) in reply to Zygo
    Zygo:
    yes, yes, I know, some Creationists don't believe in the second law of thermodynamics, but they're not exactly running server rooms are they? ;-)

    Not true. From my extensive experience reading talk.origins on Usenet, I can confirm that Creationists believe in the second law of thermodynamics, reading it as "everything's been going downhill ever since (the fall of mankind)" and that, therefore, it proves that Evilutionism, which claims some things are occasionally going uphill, is a satanic lie.

    I am not kidding!

    Don't get me, as a former student of computer science, started on what Creationists have done to information theory. Things I wouldn't do to my worst enemy. Well, you could call them "curious perversions in information theory".

    And yes, some of them arrogate themselves to measuring meaning. If only they had 1 bit of sense.

  • Morty (unregistered) in reply to Todd
    Todd:
    . . . So we would lose power to AC with the servers up and running- and when that happened it got HOT - easily 100. The only time we lost disk (or any hardware) was inside the disk cabinets after being used 24hrs a day from around the world - and it was a grand total of 5 disks we lost. I find it hard to believe a server room this small could have such a huge hardware failure, even if the AC was out for a week.
    Uh, most datacenter environments spend big bucks on cooling, and they do it for a reason: most equipment WILL crash if you have rack-dense equipment without cooling. I currently am an admin in such a datacenter environment, we sometimes have AC outages in our lab, and guess what? Equipment used to crash after a while, until we got in the habit of doing loadshedding when the AC outages were announced. Sometimes, various equipment -- including, yes, some Sun gear -- didn't come back up after an extended AC outage.

    Note that it usually takes some time at high heat before any crashes and/or damage happens. I.e. if the temperature climbs up to 100+ and then you get the AC fixed, you're probably fine. If the temperature climbs up to 100+ and you leave it like that for a couple of days, you're looking at fried gear.

    Also note that density matters more than the actual amount of equipment. I.e. you are more likely to have problems with an AC outage if you have a small amount of equipment tightly packed in a network closet than if you are in a well-ventilated equipment room with gaps between the racks and between servers mounted in the racks.

    Last, note that the article did not say that the amount of equipment in the datacenter was small, only that the amount of equipment that wouldn't come back up was small.

    I don't know if this story is true or false, but from a technical perspective, it's plausible.

  • Simetrical (unregistered) in reply to AdT
    Rhialto:
    And how is trying to conserve energy (by itself a laudable goad, albeit a bit misguided in this case) being "a liberal"?
    It's not. It's just unnecessary, seeing as how energy is always conserved anyway.
  • Simmo (unregistered) in reply to rast
    rast:
    Simmo:
    ParkinT:
    xix:

    People should pass little intelligence tests before having access to keys to such places though.

    Remember, this is a state agency. These are Government employees. It would be a violation of (some sort of) civil rights to test intelligence; as would be the case with competence. {And, probably, cause a Union hubub too}
    My we're on a real right-wing blast today aren't we.
    Griggs
    No worries. I'm sure the Supreme Court as currently constituted will soon overturn this

  • agent21 (unregistered) in reply to anne

    "if you aren't a liberal when you are 20, you don't have a heart. If you are a liberal when you are 30, you don't have a brain.

  • Jean Naimard (unregistered) in reply to Bobp0303
    Bobp0303:
    When I worked at Eaton in the early 80s, upper management all smoked.
    When I worked at $MAJOR_CIGARETTE_MANUFACTURER back in the 80’s, there was a big ashtray on top of the $MINOR_MANUFACTURER mainframe, along with a big sign that said “Thank-you for smoking”.

    $MINOR_MANUFACTURER would not dare bitch about it and replaced a washing machine every odd month or so, because they really wanted the prestigious client…

    (CAPTCHA: “poindexter”)

  • E0157H7 (unregistered)

    Something similar happened at my high school. They had a number of computer labs, but only one with Windows-based machines. They were hideous old IBM machines that desperately, desperately tried to run Photoshop, and sometimes it even worked. The room had a good 40+ machines running, and as a result they had the A/C running constantly to keep it at a comfortable temperature. One Friday, someone, for some reason, turned the A/C off and the heater on. The room was closed off from the rest of the building it was in, so the heater would have happily chugged along, oblivious to the isolated room. They also failed to shut down the PCs. It was at least 102 degrees in the lab when we came back on Monday. We spent the period hanging out in the library.

  • syadmin (unregistered)

    As a sys admin I can tell you that just about every moden server has thermal overload protection. Its a fun read, but the reality is when something starts to cook, its going to shut itself down.

  • noneofyourbusiness (unregistered) in reply to Voiceofthelost
    Voiceofthelost:
    Hey Author....

    Way to rip off my post from LiveJournal and make it sound like it's yours.

    Whoever submitted this can kiss my fat hairy ass.

    Fuckers.

    Tough luck! Deal with it n00b

  • noneofyourbusiness (unregistered) in reply to unklegwar
    unklegwar:
    all postings will be suspect for BS in my eyes.

    BS is that what you eat?

  • Phate @ Atrum Dot Org (unregistered)

    Man that's funny shit! LOL!

  • Anamouse (unregistered)

    Cute story but not real credible.

    I find it difficult to believe that the computer room did not have overtemp alarms, no State office building that I have ever heard of has an un-alarmed computer center, and most of those building have on site full time Security staff monitoring alarms.

    This employee displays obvious signs that he knows nothing about computers or computer rooms, so we can probably safely assume he doesn't work in one, so how does he know the computer room is cold if he was working somewhere else? Why would he go in there if he knows nothing about that equipment; I'm sure he wouldn't even be able to point out his email server?

    Ana

  • Frank (unregistered)

    If that story is relly true, I want a picture of that creature online if it still lives................ lol

  • Frank (unregistered)

    and: it's not the first time one can read this story. Maybe it really happend but no one does know when and where because there is no proof.

    Tja, make the story reliable or count it as fake.

  • Darklurker (unregistered)

    It's gotta be a fake/legend. If a serious server room has 3 A/C systems, then most likely those three systems aren't running off of $30 Honewell thermostats from Home Depot. They're probably on some super-complicated computerized industrial thermostat system. In which case if the guy didn't have enough sense to know why the server room needed A/C 24/7, he probably wouldn't have enough sense to know how to turn the A/C off.

  • Kent (unregistered)
    As for the employee who sent it, he decided to take an early retirement.

    He had that option?!

  • WebSphere MQ Monster (unregistered)

    Something similar happened with my company. The A/C unit was scheduled to be replaced at the data center and did not get completed that Friday when the working went home. So the job was left uncompleted and the workers left it to do Monday morning. Needless to say, by Sunday afternoon everything at the data center started failing because the temp had reached 180 degrees Celsius. A twelve hour long phone call later, someone finally said “Man, it’s really hot in the server room…” HELLO

  • Dr. Ellen (unregistered)

    Well, I have a number of comments.

    Upper Management has a key, and you're fired if you object. I worked at a museum, in charge of a climate-controlled vault full of delicate goodies. The boss insisted on having the combination to the vault. Fortunately, he couldn't work the (temperamental) lock.

    In this same museum, bean-counters put the climate control on a setback thermostat, despite evenness-of-climate being important to museum objects.

    And even the techs nod. Before I ran off and joined the museum world, I worked at a particle accelerator. Wrote the book on how to start it up on Monday evening, in fact. It had an injector that ran at 150 kV, inside a metal cage. The key that opened the cage turned the voltage off. Which is why I was so startled when it spat lightning bolts at me one evening. Somebody had been working on it, had disabled the safeties, and hadn't set them going again at the end of the day. Something similar happened at Chernobyl, as I recall, only worse.

    I don't know if this particular incident actually happened this particular way, including the self-congratulatory, scolding e-mail. But I can remember sufficient elements of the incident in my own life to make it eerily plausible, up to and including the self-congratulation and the fact the perp got off light because (s)he was Management and had Good Intentions.

  • James C (unregistered)

    Wow, truly a WTF if ever I saw one. Fastest way I can think of to annihilate a server room.

    One question, though...

    When did saving energy become "liberal" or a bad thing? Isn't it just good behaviour? Why are people here upset about that?

    Not switching off the A/C in a server room, of course.

  • rjschwarz (unregistered) in reply to akatherder
    akatherder:
    whicker:
    No, I know you people just purchase whatever is available, and have little if any say in the parts that go into the equipment. But by now, things have got to be perfected to the point where you can add a server farm to your shopping cart and have it arrive next day.

    It also depends what vendor. If you order a Sun server with 4 GB of memory and add 4GB more, they put it in another cardboard box and ship it along with the server.

    If you buy a Sun Blackbox server you just plop it into an wharehouse and plug water and power into the it and it is supposed to work. Not far from your server farm in a shopping cart although I suspect the price tag will give you sticker shock.

  • Blacque Jacques Shellacque (unregistered)

    Bwaaahahahahahahahahaaaa!!!!!!!

  • Anonymous (unregistered)

    H. Beam Piper wrote a short story that explains just why these things happen. And what's scary, it's SUPPOSED to be fiction, but it's gonna happen somewhere.

    "The Day of the Moron"

    http://www.gutenberg.org/files/18949/18949-8.txt

  • AdT (unregistered) in reply to agent21
    agent21:
    "if you aren't a liberal when you are 20, you don't have a heart. If you are a liberal when you are 30, you don't have a brain.

    And if you belieed Bush's lies about Iraqi WMDs, then you have the intelligence quotient of an amoeba.

  • AdT (unregistered) in reply to Simetrical
    Simetrical:
    Rhialto:
    And how is trying to conserve energy (by itself a laudable goad, albeit a bit misguided in this case) being "a liberal"?
    It's not. It's just unnecessary, seeing as how energy is always conserved anyway.

    Since no one has mentioned the third law of thermodynamics yet, let me point out that too much cooling is not good for the servers either:

    As a system approaches absolute zero, all processes cease...

  • network guy (unregistered)

    WOW that's messed up but what I find more disturbing is that primary, secondary and tertiary environmental units where shutdown and no one was alerted? someone really dropped the ball on monitoring I mean it's only the State of NY right? Way to be on top of your gear and way to maintain security.

    network guy

  • Matt (unregistered) in reply to WebSphere MQ Monster
    WebSphere MQ Monster:
    Needless to say, by Sunday afternoon everything at the data center started failing because the temp had reached 180 degrees Celsius.
    Tech walks into data center... it's 180 degrees Celsius...

    Holy crap it's--

    His blood boils. He crumples to the floor.

    ...hhhhhhhhhhoo....t.

  • Bryan (unregistered)

    This story caused me physical pain. Ooohhh! Ohh.

  • Bryan (unregistered)

    We had a massive air-conditioning outage in the server room at work once. The IT guys ran downstairs and just started shutting everything off. I think we did ok, hardware-wise. I just played darts that day. It was all good.

  • Help Desk Admins (unregistered)

    Actually, that must have been a someone who works on the corporate hallway. Only your extremely educated, hight paid idiots could be that STUPID and CLUELESS

  • Arlo (unregistered)

    This story is obviously fake but accurate.

  • rjsasko (unregistered)

    I don't know why so many think this is an urban legend because similar stupidities happen all of the time. A couple of years ago a rather expensive steel mill in Chicago was in bankruptcy and a Trustee was appointed to manage the plant. The Trustee kept getting a gas bill for the closed-down plant. Since his job was to minimize costs and maximize the value of the property he had the gas shut off since it obviously wasn't needed for a shuttered steel mill. The gas was used to keep the special refractory bricks warm. Once they cool they will crack and turn to dust. Scratch one very expensive blast furnace.

  • Mongo (unregistered) in reply to BRus

    Yeah, I don't know if this whole thing is real or not, I suspect it's based on something that actually happened and just been embellished but I can totally see it happening. I worked at a server farm a couple of years ago, back when energy prices were skyrocketing in California we had a facility manager decide that all AC would be shut down over the weekend, turned off Friday night starting at 6PM and not turned on again until 8AM Monday morning. No notes went out, no meetings were held, no one knew about it. Certainly not us guys on the server farm.

    So he implements the policy and the very first weekend the temps outside were pushing 100°, the on-call person didn't take his pager with him when he left for the weekend, neither did our boss, so by the time the escalation pages hit it was the second line manager who was calling people up to find out what was going on. I walked into the server room at about 8PM on a Saturday night, it was still a balmy 94° outside, and it was close to 106° in the server room. We had a half dozen servers that had shutdown because of the heat, and three more that were failing. I had to call up the facility manager to get his approval to turn on the AC in the server room so we could get the room temp down. The long term exposure to the heat actually damaged components on three of the servers, cost us nearly $4000 plus the money we lost due to missed SLAs, pretty much wiped out the cost savings from shutting down the AC.

  • Larry O (unregistered)

    I work in the government and we have all of our Network/server rooms keycarded to IT only. And any one stupid enough to leave access cards in the desk....

    We also have triple redundant warnings that the AC is not working or has been turned off. The error msgs start at 68 Farenheit and don't stop until the temp falls back to 66 Farenheit.

  • cessnadav (unregistered) in reply to Zygo

    Hmmmm. First, 2nd law does not apply to the creation/evolution debate since neither creation nor evolution are thermodynamic processes. Sorry, kids, it's laziness, not entropy, that makes your room get dirtier the more you don't clean it. Second, cooling efficiency doesn't apply either because the servers use FANS to cool them. Air conditioners use thermodynamic cycles that become less efficient as the outdoor temperature rises, because they depend on heat rejection to your outdoor unit, which obviously doesn't happen as well when it's hotter outside. Fans just blow air that is hopefully cooler than what you're trying to cool...

  • armed liberal (unregistered) in reply to Mr Steve

    Y'know I'm pretty sure the Original version on livejournal, since deleted; was at a private company, not a state office...

    But, hey have

  • tomliotta (unregistered) in reply to pinball
    pinball:
    PyroTyger:
    Fine, Urban Legend, okay, but this is supposed to be a *state* department? I've worked in government long enough to know that there is always, *always* someone in.
    You must work for a different government. My sister works for the government as *the* IT department (yes, she is the only IT person in the whole office). I can affirm that you are wrong in that when she goes on vacation, has a 3-day weekend, leaves for the day or whatever, there is not somebody there to watch over the stuff.
    I was gonna skip posting, but saw this one. I worked municipal, county and state positions for around 15 years; last time was approx 10 years ago. I don't recall any stretch where a significant server could be down for more than a couple hours at any time of any day without IT being called.

    However, don't forget what the story said... The dolt was in the office on Monday of a 3-day weekend. No indication was given of what time of day the A/C was turned off. Coulda been 8PM Monday night. Coulda even been other staff working in the same building Saturday, Sunday and part of Monday. Coulda even been some IT staff at times.

    Regardless, the dolt was there; so we know that someone worked during off-hours.

    IF the story has any truth, there might not have been outages until 5AM Tuesday morning.

    Even so, the last state agency I worked for was approx 10 years ago. Maybe a data center a third the size described in the article. We had water, power, smoke, noise and temperature sensors installed a good 5 years before I left. No e-mail nor paging -- it was direct into the phone system on separate power. Anybody could call it to get a status or simply listen to whatever noises were happening. (And the infrequent citizen would from time to time.)

    Round-robin dialing to every IT member. Whoever was on-call first on the list for a week was responsible for changing the call-sequence for the next week to move themselves from the top to the bottom of the sequence or spend another week at the top. Simple, reliable, and nowhere near expensive enough to be concerned over.

    But a State Tax Commission? I have no idea how much communications traffic must be going on, just to other State Departments if not to various outside locations. Hard to imagine circumstances that would let this cascade as described.

  • chsw (unregistered)

    In which state did this happen? Which agency?

    chsw in NY

  • Muchsake (unregistered) in reply to AdT

    I have seen the film of bush reading that book. Have you ever worked in a kindergarten or children's library? If the cover is loose then it has a 50% chance that it is upside down. The cover being upside down does not necessarily mean that the book is.

  • Anonymous Nerd (unregistered)

    It doesn't matter if this particular story is fake or not. It's funny! This crap happens all the time. Stupid hippies. Go put your Birkenstocks back on and shut up.

  • turning thirty. (unregistered) in reply to agent21
    "if you aren't a liberal when you are 20, you don't have a heart. If you are a liberal when you are 30, you don't have a brain.

    omg... it makes so much sense now.

  • The Commenter (unregistered) in reply to Ares
    Ares:
    Al Gore?

    CAPTCHA - onomatopoeia - almost made me not want to comment

    What kind of a dumb fk moron fkhead are you that keeps putting your captcha string in.

    Sorry to lower the tone of the posts but I know you were all dying for somebody to say it.

Leave a comment on “I'm Sure You Can Deal”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article