• Rboy (unregistered) in reply to Edd

    It really was the neo-cons that caused the 'accident'.

  • Bob N Freely (unregistered) in reply to Hans
    Hans:
    What would Chunk Norris do?

    The truffle shuffle.

  • (cs)

    Years ago I worked for a 911 dispatch agency, and just before I left, we began preparations for relocating the center. It was an exercise in frustration. In addition to management's obvious assumption that they could just back a truck up to the door and load up desks, and have the dispatchers report to the new place the next morning, we had these technical marvels:

    1. Carpet in the computer room
    2. Redundant telephone trunks (which is good)... that were installed in the same underground trench... not so good
  • Autocracy (unregistered) in reply to Lars Vargas

    I gotta say, I don't think Halon, sprinklers, or even an air-free environment would have done a damn thing to save those machines.

  • Herby (unregistered)

    This whole incident gives new meaning to the term "Firewall".

    You can all groan now.

  • dude (unregistered) in reply to lonewolf

    What would you suggest they use to clean up the mess?

  • James (unregistered) in reply to AccessGuru
    AccessGuru:
    What would Jesus do?

    He's got it covered: Jesus Saves.

  • Andrew (unregistered) in reply to Lars Vargas
    Lars Vargas:
    TRWTF: No computer-safe fire suppression appears to have been in place (or at least not mentioned). Halon, anyone?

    And yeah, what about those off-site backups?

    They had pressurized acetylene tanks next to the computers. The explosion opened the building to the air. Inert gas or foam fire suppression systems might not have stopped it.

    In general, store explosive items away from anything valuable. This includes harmless fertilizer, also known as ammonium nitrate.

  • (cs)

    Hi, it's my story/event, so I'll try to answer a few questions:

    No, unfortunately, there aren't any other pictures, that one was taken with my crappy VGA camera phone that night, the closeup pics are all black. The following day they blocked off the whole area, as someone decided that a gaping hole in a load-bearing wall probably wan't helping the building stay up.

    what about those off-site backups?
    Those aren't much good if you a lack a server to put them on. We did use them once we had jurry-rigged a network and replaced half a dozen routers to get it work a little. They were stored in the same building, but at least they were stored in "the vault" in the basement. I've never actually seen "the vault", so I can only hope it wasn't designed by the person who designed the server room.
    Seriously, I'd be happy about getting a couple racks full of new toys.
    Oh, that was nice. The installing of said racks in a temporary building, fixing/replacing the destroyed cables followed by round-the-clock work to get everything working, then doing it again when the new building was done, wasn't so much fun.
    Why were there a dozen acetylene tanks in a server room?
    Well, they weren't stored in the same room. They were stored in the nearby gas storage building. (on the other side of the hallway, meaning one concrete wall, and a brick interior wall.) Except that it wasn't actually meant to store gas cilinders, it was designed to store sand, cement, gravel sheet metal and the like. It had, to quote one of the civil enginering students, "More in common with a lean-to garden shed then a gas bunker".
    What would you suggest they use to clean up the mess?
    A wrecking ball and a bulldozer. The whole building had to go, seeing how there was a hole in the wall large enough to drive a small car through.
  • (cs) in reply to Cyrus
    Cyrus:
    JohnnyWalls:
    Fist?

    Congratulations on making a "fist" comment 25 minutes after the first comment?

    That is how slow people read these days.

  • synp (unregistered) in reply to Herby
    Herby:
    This whole incident gives new meaning to the term "Firewall".

    You can all groan now.

    Actually, that's the old meaning

  • KosMot (unregistered) in reply to AccessGuru

    Don't you know that Jesus Saves ... and stores the backup offsite.

  • elvis (unregistered)

    their firewall was obviously not up to the job.

  • Bobbo (unregistered) in reply to Man 987876980
    Man 987876980:
    Ben:
    Hans:
    What would Chunk Norris do?
    Chuck Norris would use a dozen acetylene tanks to create an explosion which would rip through four concrete walls and reduce an entire data center to slag and ash. HTH
    No, not Chuck Norris, he said ChuNk Norris... who I presume is Chuck's obese brother.

    Yeah, ChuNk Norris would probably sit around eating a Mars Bar. Sweating. In a decidedly non-action packed manner.

  • mizchief (unregistered) in reply to Mutant
    Mutant:
    Lars Vargas:
    And yeah, what about those off-site backups?

    They may have had offsite backups... but without a server to restore them to, it's still going to be a lot of work...

    It's also really hard for people to see ahead to what effects a real disaster will bring before it happens. Especially those signing the bills. Who's really going to predict a server room blowing up?

    My company has suddenly bumped DR right to the top of the priority list after 3 incredible near misses (all unrelated) in 2 weeks. Before then enhancing our current (fairly reasonable) strategy wasn't deemed cost effective. Probably fair enough... I mean, how far do you take it? Nuclear War? Asteriod Strike? The Second Coming?

    My rule of thumb is to protect data against all Disasters from which the company as a whole could survive. In a company of less than 20 you would probably be able to recover your business after a fire, but not so much a direct nuclear attack. But a large corporation with offices all over the world, the company could survive such an attack so they should backup the data in each office far away from it's location.

  • (cs) in reply to alcari
    alcari:
    Hi, it's my story/event, so I'll try to answer a few questions:

    No, unfortunately, there aren't any other pictures, that one was taken with my crappy VGA camera phone that night, the closeup pics are all black. The following day they blocked off the whole area, as someone decided that a gaping hole in a load-bearing wall probably wan't helping the building stay up.

    One's enough. It's clear that you had an impressively devastating problem. That said, it would have been awesome to see the concrete embedded in the computers.

    what about those off-site backups?
    Those aren't much good if you a lack a server to put them on. We did use them once we had jurry-rigged a network and replaced half a dozen routers to get it work a little. They were stored in the same building, but at least they were stored in "the vault" in the basement. I've never actually seen "the vault", so I can only hope it wasn't designed by the person who designed the server room.

    To most people, a catastrophic failure is when the power supply shorts out and takes the motherboard with it. Planning for complete physical destruction simply never enters most people's minds.

    One place I worked had no backups whatsoever. Eventually they started contracting that out and have offsite backups once a day.

    Another place had offsite backups done twice a day.

    Except that it wasn't actually meant to store gas cylinders, it was designed to store sand, cement, gravel sheet metal and the like. It had, to quote one of the civil engineering students, "More in common with a lean-to garden shed then a gas bunker".

    Indeed. The university is bloody lucky they aren't looking at a wrongful death lawsuit. You're unbelievably fortunate that you weren't in there when the tanks exploded.

    I'd submit this to the RISKS digest. Multiple fatalities were only avoided by luck and timing.

    What would you suggest they use to clean up the mess?
    A wrecking ball and a bulldozer. The whole building had to go, seeing how there was a hole in the wall large enough to drive a small car through.

    Yeah, I would have suggested a bulldozer too. There's nothing salvageable in a building that's been exposed to an explosion like that. There would be serious weakening of the structural components.

  • Edward Royce (unregistered) in reply to floorpie
    floorpie:
    That's nothing,

    I worked at a place where the server room was under the bathroom from the floor above... until the neighbours shower sprung a leak...

    Amusing isn't it?

    I worked at a place that had a hermetically sealed server room completely enclosed in glass. Very attractive, very contemporary. With a huge 20+" water main running right through the server room, glass enclosures and all, from one side all the way through to the other.

    I had a lot of questions about that.

  • (cs) in reply to Mutant
    Mutant:
    I mean, how far do you take it? Nuclear War? Asteriod Strike? The Second Coming?

    I worked at a company that was prepared for those scenarios. They would send a CD with the latest source code to an off-site facility built located deep within some mountain. They had an agreement with their clients that, in the event NYC was vaporized, this company would send out copies of the source code and the clients could continue to develop the software themselves. I guess it's too bad the software didn't really work in the first place...

  • LEGO (unregistered) in reply to Edward Royce
    Edward Royce:
    floorpie:
    That's nothing,

    I worked at a place where the server room was under the bathroom from the floor above... until the neighbours shower sprung a leak...

    Amusing isn't it?

    I worked at a place that had a hermetically sealed server room completely enclosed in glass. Very attractive, very contemporary. With a huge 20+" water main running right through the server room, glass enclosures and all, from one side all the way through to the other.

    I had a lot of questions about that.

    That is an aquarium waiting to happen. ;-)

    -Lego

  • Christian (unregistered) in reply to Outlaw Programmer
    Outlaw Programmer:
    Mutant:
    I mean, how far do you take it? Nuclear War? Asteriod Strike? The Second Coming?

    I worked at a company that was prepared for those scenarios. They would send a CD with the latest source code to an off-site facility built located deep within some mountain. They had an agreement with their clients that, in the event NYC was vaporized, this company would send out copies of the source code and the clients could continue to develop the software themselves. I guess it's too bad the software didn't really work in the first place...

    This is called Code Escrow. A lot of "bigger" companies see it as an entry requirement for doing business with a smaller company. I believe the small company I worked with had to put source for each release into code escrow for 25 years in case a customer needed it. Often, each customer would have their own escrowed code- it was worth it to get the bigger license deals.

  • (cs) in reply to lonewolf
    lonewolf:
    TRWTF is that shovel in the server room..

    And how do you deal with the daily bullsh*t the users give you? ;-)

  • bla (unregistered) in reply to James
    James:
    AccessGuru:
    What would Jesus do?

    He's got it covered: Jesus Saves.

    Buddha does incremental backups ;-)

    http://www.syswear.com/view/tshirts?d=46

  • Maarten (unregistered) in reply to Edward Royce
    Edward Royce:
    floorpie:
    That's nothing,

    I worked at a place where the server room was under the bathroom from the floor above... until the neighbours shower sprung a leak...

    Amusing isn't it?

    I worked at a place that had a hermetically sealed server room completely enclosed in glass. Very attractive, very contemporary. With a huge 20+" water main running right through the server room, glass enclosures and all, from one side all the way through to the other.

    I had a lot of questions about that.

    Questions such as: "where are the fish?"

  • iToad (unregistered)

    The real WTF is that somebody stored about a dozen acetylene tanks inside a building. Acetylene gas will spontaneously explode when compressed, so the cylinders are actually filled with acetylene dissolved in acetone.

    When the cylinders split open, the resulting fire was fueled by both the acetylene, and a lot of acetone. It's not suprising that the entire building burned up.

    As a young lad, I actually saw an acetylene cylinder cook off in a fire. The blast knocked the firemen down, and broke windows two blocks away.

    Like the man said, always chain up your gas cylinders.

  • (cs) in reply to AccessGuru
    AccessGuru:
    What would Jesus do?

    Jesus saves. The servers, however, still would have taken full damage.

    [/ripped off joke]

  • Jeff (unregistered)

    Clearly, you needed a more secure firewall.

  • florida (unregistered) in reply to Mutant
    Mutant:
    It's also really hard for people to see ahead to what effects a *real* disaster will bring before it happens. Especially those signing the bills. Who's really going to predict a server room blowing up?

    that's why here in hurricane land we have a remote hot site a hundred miles away. it has a dedicated line to keep data synced. so if our building collapses, our failover server kicks on and "relatively seamlessly", our network is back online.

  • Lars Vargas (unregistered) in reply to Grovesy
    Grovesy:
    Lars Vargas:
    TRWTF: No computer-safe fire suppression appears to have been in place (or at least not mentioned). Halon, anyone?

    And yeah, what about those off-site backups?

    umm, unless my sarcasm detector has failed, I don't see how Halon would have helped against an explosion which embedded bits of concrete into the servers...

    Silly me! I assumed they were hardened servers.

  • a former big-fiver (unregistered) in reply to floorpie
    floorpie:
    That's nothing,

    I worked at a place where the server room was under the bathroom from the floor above... until the neighbours shower sprung a leak...

    I once worked in a courthouse with a small internal jail in it (for holding folks going to/from their hearings). The datacenter was two floors below the jail. A nice, secure, Halon equipped datacenter with good gear in it. Until the day the inmates plugged the toilets & deliberately overflowed them. Wiping sh*t off hundreds of thousand of dollars' worth of servers gives new meaning to 'having a crappy day'.

  • Volpone (unregistered)
    One grim-faced coworker cleared his throat, and walked over to the remains of a less-fortunate metal shelf. He brushed off some soot from a small, metal plate. It read "Secondary Server - Do Not Touch."
    "You fool, it said not to touch it! Now we'll never get our data back!"
  • esse (unregistered)

    This isn't a WTF, it is insurance fraud to get new server equipment + new data center.

    I'm sure they'll pick a prime location next to the Los Alamos missile testing range.

  • Winslow Theramin (unregistered) in reply to alcari

    Any dates and locations so we can verify this against police/fire reports? Or is this just the usual made-up Daily WTF crap?

  • (cs)

    Reminds me of this video from HP: http://www.hp.com/go/DisasterProof In which they blow up a simulated data center "real good."

    It seemed pretty contrived when I first saw it, but maybe I was wrong.

  • jonny s. (unregistered) in reply to Winslow Theramin
    Winslow Theramin:
    Any dates and locations so we can verify this against police/fire reports? Or is this just the usual made-up Daily WTF crap?

    Are you guys hiring? I want a job where I don't have to do anything so I can pursue my true dream of disproving THE ENTIRE INTERNET.

  • Bill Lumbergh (unregistered) in reply to Lars Vargas
    Lars Vargas:
    Grovesy:
    Lars Vargas:
    TRWTF: No computer-safe fire suppression appears to have been in place (or at least not mentioned). Halon, anyone?

    And yeah, what about those off-site backups?

    umm, unless my sarcasm detector has failed, I don't see how Halon would have helped against an explosion which embedded bits of concrete into the servers...

    Silly me! I assumed they were hardened servers.
    Annealing is part of the hardening process.

  • nulla (unregistered) in reply to Winslow Theramin
    Winslow Theramin:
    Any dates and locations so we can verify this against police/fire reports? Or is this just the usual made-up Daily WTF crap?
    The fake is a lie.
  • Mark Richards (unregistered) in reply to iToad

    Acetylene is incredibly unstable. It can explode when being compressed. It can explode if you decompress it too quickly (i.e. open the tank too quickly or use too large a nozzle- that can cause the acetylene to bubble out of the acetone and when that happens- it's usually time to run).

    Acetylene is not just dissolved in acetone- the tanks it is stored in also contain a porous filling (Agamassan) that helps stabilize it.

    Keeping that many acetylene tanks in one place probably isn't even legal unless you have a storage facility designed for them. The reasons are obvious.

  • Anom (unregistered) in reply to Mutant
    Mutant:
    Lars Vargas:
    And yeah, what about those off-site backups?

    They may have had offsite backups... but without a server to restore them to, it's still going to be a lot of work...

    It's also really hard for people to see ahead to what effects a real disaster will bring before it happens. Especially those signing the bills. Who's really going to predict a server room blowing up?

    My company has suddenly bumped DR right to the top of the priority list after 3 incredible near misses (all unrelated) in 2 weeks. Before then enhancing our current (fairly reasonable) strategy wasn't deemed cost effective. Probably fair enough... I mean, how far do you take it? Nuclear War? Asteriod Strike? The Second Coming?

    The only thing you have to worry about with The Second Coming is your magnetic tapes anyway.

    /Obscure Futurama Reference

  • Anom (unregistered) in reply to Mutant
    Mutant:
    Lars Vargas:
    And yeah, what about those off-site backups?

    They may have had offsite backups... but without a server to restore them to, it's still going to be a lot of work...

    It's also really hard for people to see ahead to what effects a real disaster will bring before it happens. Especially those signing the bills. Who's really going to predict a server room blowing up?

    My company has suddenly bumped DR right to the top of the priority list after 3 incredible near misses (all unrelated) in 2 weeks. Before then enhancing our current (fairly reasonable) strategy wasn't deemed cost effective. Probably fair enough... I mean, how far do you take it? Nuclear War? Asteriod Strike? The Second Coming?

    The only thing you have to worry about with The Second Coming is your magnetic tapes anyway.

    /Obscure Futurama Reference

  • notme (unregistered) in reply to Anom
    Anom:

    The only thing you have to worry about with The Second Coming is your magnetic tapes anyway.

    /Obscure Futurama Reference

    I have seen all of the Futurama series, but I can't recall that bit. In what episode did this happen?

  • Anom (unregistered) in reply to notme
    notme:
    Anom:

    The only thing you have to worry about with The Second Coming is your magnetic tapes anyway.

    /Obscure Futurama Reference

    I have seen all of the Futurama series, but I can't recall that bit. In what episode did this happen?

    Season 1 - Episode 12 "When Aliens Attack"

    The Omicronians demand that Single Female Lawyer be put back on the air and Farnsworth explains that most cassette tapes were erased during the Second Coming of Jesus in 2443.

    The fact that I remembered all of this (except the episode number and the year of the second coming) off the top of my head might mean I have watched too much Futurama, if that's possible.

  • Mrs Post (unregistered)

    We had a minor incident at a company I worked at once.

    Of course there was a drain under the raised floor in case the sprinklers went off. Only logical.

    Well, one day the waste line for the company decided to back up. The company had to use the shuttle bus to take people back and forth to the nearest gas station for 'convenience breaks' while they figured out if this could be fixed.

    (Savvy people see where this is going.)

    Well, whoever designed the building and put the drain under the computer room raised floor forgot one small thing. A one-way flow valve.

    I get the call from the network wranglers to get everyone in there to shut down servers as the server room was flooding. A flying V formation of techs was mobilized and all the servers were being shut down as fast as they could be while the 'water' was rising.

    Please note the quotes around the word water. It wasn't just water that was flooding the server room. Waste line, remember?

    The 'water' was getting too close for comfort to the power junction. Things got shut down and eventually the 'water' stopped just 1/4 inch (6.5mm for you metric folks) shy of the power.

    Fans and a long orange extension cable restored limited connectivity to the mainframe.

    Overall we were down for about a day. I truly feel sorry for the emergency crew on call to clean up the mess under the floor. One of the network wranglers couldn't take it when he saw things floating.

    Even better, all the network wranglers were housed in the server room.

    Ah, the good old days.

  • (cs) in reply to dgvid
    dgvid:
    Reminds me of this video from HP: http://www.hp.com/go/DisasterProof In which they blow up a simulated data center "real good."

    It seemed pretty contrived when I first saw it, but maybe I was wrong.

    If the marketing department had too much power, I could see this happening:

    HP: We destroyed a large office building with a low yield nuclear warhead to show how are data center will fail over.

    Person: Couldn't you have just cut the power?

    HP: Well, yeah, but it wouldn't have been as cool.

  • fava (unregistered)

    Where I work the server room in one building was expanded into a room that was obviously never envisioned to be a server room. There is a 8" diameter chilled water line running through the center of the room. If that line breaks there will be a foot of water in the room no time flat. But we did install drip trays so that "minor" leaks could be controlled.

  • Kay B (unregistered) in reply to dkf

    I have a backup of my name on mars, does that count? :P The backup media is a DVD sitting on Phoenix, somewhere in Green Valley of Vastitas Borealis, Mars.

  • (cs) in reply to Mark Richards
    Mark Richards:
    Acetylene is incredibly unstable. It can explode when being compressed. It can explode if you decompress it too quickly (i.e. open the tank too quickly or use too large a nozzle- that can cause the acetylene to bubble out of the acetone and when that happens- it's usually time to run).

    Acetylene is not just dissolved in acetone- the tanks it is stored in also contain a porous filling (Agamassan) that helps stabilize it.

    Keeping that many acetylene tanks in one place probably isn't even legal unless you have a storage facility designed for them. The reasons are obvious.

    OK, We're all being incredibly dense here. I know I am.

    What, exactly, is the point of storing half a dozen acetylene tanks anywhere near anything else?

    Or is the OP just anonymised beyond chemical belief?

    "Dude ... hee hee hee ... we managed to hide an entire crystal meth factory behind the back-up server before the sprinkler went on for an automatic test and we forgot to shut off the peroxide feed ... hee hee hee..."

    Or are we all missing something here?

  • (cs) in reply to SeekerDarksteel
    SeekerDarksteel:
    AccessGuru:
    What would Jesus do?

    Jesus saves. The servers, however, still would have taken full damage.

    [/ripped off joke]

    Badly ripped off, I believe.

    The original is "... and Kenny Dalgliesh scores on the rebound."

    For ignorant Yanks, substitute Michael Jordan or Magic Johnson for Dalgliesh.

  • (cs) in reply to alcari
    alcari:
    Why were there a dozen acetylene tanks in a server room?
    Well, they weren't stored in the same room. They were stored in the nearby gas storage building. (on the other side of the hallway, meaning one concrete wall, and a brick interior wall.) Except that it wasn't actually meant to store gas cylinders, it was designed to store sand, cement, gravel sheet metal and the like. It had, to quote one of the civil engineering students, "More in common with a lean-to garden shed then a gas bunker".
    Sorry, alcari, I missed that.

    Mind you ... sand, cement, gravel, sheet metal, small rodents, massively explosive devices under extreme, high-temperature pressure?

    Well, I've dealt with bosses who resemble small rodents before. Possibly they're under extreme, high-temperature pressure.

    That's Boyle's law for you.

  • (cs) in reply to real_aardvark
    real_aardvark:
    SeekerDarksteel:
    AccessGuru:
    What would Jesus do?

    Jesus saves. The servers, however, still would have taken full damage.

    [/ripped off joke]

    Badly ripped off, I believe.

    The original is "... and Kenny Dalgliesh scores on the rebound."

    For ignorant Yanks, substitute Michael Jordan or Magic Johnson for Dalgliesh.

    No it wasn't "ripped" off. You are quoting a hockey jock (HOW the HELL do you put a basketball player into a hockey joke? There are no "saves" in basketball)

    But this "ripped" off joke refers to a different game.

  • (cs) in reply to akatherder
    akatherder:
    Why were there a dozen acetylene tanks in a server room? Do they keep some dynamite in the phone closet?

    Did you actually READ the article? I'm just going to take a wild guess that the server room wasn't the only room in the building.

Leave a comment on “A Bit More Dire”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article