• Andreas Christodoulou (unregistered)

    And I know of a few IT Professionals who would STILL just say "Ah, let's just reboot the affected servers then" when told of this problem.

  • (cs)

    I'd like to hear the support calls... "Hi, I can't login" "Yeah, the server room blew up" "What?" "It blew up. Someone tried to log in during maintenance even though we told everyone not to do it. Say... was that you trying to log in a few minutes ago?" click

  • (cs)

    So the server room burns down, and there's no offsite backup?

    This is what counts as a "curious perversion in information technology" nowadays?

  • Dutchy (unregistered)

    "At least we have backups securely stored." "You mean.. this?"

    • Holds up a sign: "Backup shelf, do not touch."
  • (cs) in reply to Andreas Christodoulou
    Andreas Christodoulou:
    And I know of a few IT Professionals who would STILL just say "Ah, let's just reboot the affected servers then" when told of this problem.

    If you can't turn it off and on again, they won't even talk to you.

  • Edd (unregistered)

    How come the box of copier paper survived unscathed?

  • alx5000 (unregistered)

    Core coment Do Not Touch

  • Resolute (unregistered) in reply to alx5000
    alx5000:
    *Core coment* Do Not Touch
    1. Read article
    2. Parrot something said in article
    3. ???????
    4. Featured comment!
  • JohnnyWalls (unregistered)

    Fist?

  • Edwin (unregistered)

    Also the smartest place to keep your second server if you planned its for cases like this. Why people always think of things like that when its to late.

  • (cs) in reply to NaN

    Reminds me of a story from a friend,

    He was brought in towards the end of the procurement procedure for a disaster recovery contract his company was purchasing; it was pretty big they would install a complete duplicate of their live + staging systems in a DR centre.

    The very senior managers had done all their stuff, selected the vendor that gave them the most freebies along with the cheapest price and it was up to him to get on with the actual ‘doing’ part of the deal.

    After preliminary contract acceptance and various statements of intent, he was down at his company’s data centre, on the phone to his counterpart underlings from the DR company, when out of the corner of his eye, out through the window, across the way and noticed a large, squat building in the same business park with the DR companies logo on the front. He politely asked where their data centre was...

    The real WTF was, management ignored his 'concerns' the contract went ahead as the DR vendor was the cheapest.. shame the DR centre was pretty much next door.

  • (cs)

    TRWTF: No computer-safe fire suppression appears to have been in place (or at least not mentioned). Halon, anyone?

    And yeah, what about those off-site backups?

  • (cs) in reply to JohnnyWalls
    JohnnyWalls:
    Fist?

    Congratulations on making a "fist" comment 25 minutes after the first comment?

  • (cs)

    Well, on the bright side they get the chance to test out the procedures for worst case scenario.. assuming the procedure manual survived.

  • (cs) in reply to Lars Vargas
    Lars Vargas:
    TRWTF: No computer-safe fire suppression appears to have been in place (or at least not mentioned). Halon, anyone?

    And yeah, what about those off-site backups?

    umm, unless my sarcasm detector has failed, I don't see how Halon would have helped against an explosion which embedded bits of concrete into the servers...

  • floorpie (unregistered)

    That's nothing,

    I worked at a place where the server room was under the bathroom from the floor above... until the neighbours shower sprung a leak...

  • (cs)

    What would Jesus do?

  • (cs) in reply to Shial
    Shial:
    Well, on the bright side they get the chance to test out the procedures for worst case scenario.. assuming the procedure manual survived.

    No worries, it was kept in PDF format on the server hard drive.

    And in case something happened to it, you'll find another copy in the /backups/ directory of the same drive...

  • Quango (unregistered)

    Reminds me of an incident a few years ago when I was tech support for a "large international merchant bank" in the City of London, when the Bishopsgate bomb went off (see http://en.wikipedia.org/wiki/1993_Bishopsgate_bombing )

    Fortunately our building only suffered a few cracked and broken windows - but was just inside the police cordon so all of the several thousand staff were evicted the whole site was off-limits for about a day (a LONG time if you're a market trader).

    The nasty realization for the disaster recovery team was that the Disaster Recovery (DR) site, one of the bank's older unused offices, was right in the middle of the cordon. Worse still, the main fibre optics linking the two sites ran right down the Bishopsgate. Which now sported a "bomb crater about 4.3 m deep and 4.5 m in diameter"

    Needless to say our DR site moved elsewhere later that year.

  • (cs)

    Well, at least nobody can blame the tech support team for that one, though I'm sure some network guru is having an apoplectic stroke over the primary and backup servers in physical proximity by now.

  • Mike (unregistered)

    Perhaps it was the nature and content of the article, but I swear when I read the last part about the secondary server, I thought it read "Secondary Server - Do Not ToRch".

  • (cs)

    TRWTF is that shovel in the server room..

  • Matt (unregistered) in reply to WhiskeyJack

    You can solve any computer problem by adding more memory.

  • (cs) in reply to AccessGuru
    AccessGuru:
    What would Jesus do?
    Keep off-Earth backups, of course. You can't be too careful.
  • Hans (unregistered)

    You know, I just went through this with about 7500 hundred other customers at the planet, when there was an electrical explosion that knocked the entire center off line for 2 days. 9000 servers went dark instantly.

  • Hans (unregistered) in reply to AccessGuru

    What would Chunk Norris do?

  • Nobody (unregistered)

    I was waiting for the molten slag of a shelf with a tag reading "Backup Tapes - Do not Touch".

  • (cs)

    Why were there a dozen acetylene tanks in a server room? Do they keep some dynamite in the phone closet?

  • (cs) in reply to Shial
    Shial:
    Well, on the bright side they get the chance to test out the procedures for worst case scenario.. assuming the procedure manual survived.
    You mean this? (Holding up charred remains of a cover sheet, on which you can still make out fragments of words "...nual - Do No T...")
  • Ghost (unregistered) in reply to dkf
    dkf:
    AccessGuru:
    What would Jesus do?
    Keep off-Earth backups, of course. You can't be too careful.

    In case of the Vogons arriving?

  • (cs)
    [image]

    So that's why it's a bomb.

    Given the description of these canisters, looks like the Atari ST Row of Bombs might be more appropriate...

    [image]
  • Steve (unregistered)

    This isn't a WTF?. It's an [b]Oh, sh*t[/i].

    On the bright side, it means no more struggling with that crappy old server. . .

  • f0dder (unregistered) in reply to AccessGuru
    AccessGuru:
    What would Jesus do?
    Jesus... wept.
  • Hoser, Eh (unregistered)

    certainly not best practices, then.

  • drex_ej (unregistered) in reply to Steve

    Seriously, I'd be happy about getting a couple racks full of new toys. Then again, we have off site storage.

  • CoderHero (unregistered) in reply to Steve
    Steve:
    This isn't a WTF?. It's an [b]Oh, sh*t[/i].

    On the bright side, it means no more struggling with that crappy old server. . .

    Perhaps not THAT crappy old server, but I can see this scenario:

    Manager 1: That fire cost us $120,000 in computer equipment and we don't have the budget to replace it Manager 2: Well, I did hear that the Salvation Army is selling used 486's on the cheap!

  • Mutant (unregistered) in reply to Lars Vargas
    Lars Vargas:
    And yeah, what about those off-site backups?

    They may have had offsite backups... but without a server to restore them to, it's still going to be a lot of work...

    It's also really hard for people to see ahead to what effects a real disaster will bring before it happens. Especially those signing the bills. Who's really going to predict a server room blowing up?

    My company has suddenly bumped DR right to the top of the priority list after 3 incredible near misses (all unrelated) in 2 weeks. Before then enhancing our current (fairly reasonable) strategy wasn't deemed cost effective. Probably fair enough... I mean, how far do you take it? Nuclear War? Asteriod Strike? The Second Coming?

  • some guy (unregistered)

    If (User.Name == "Lyle") { Server.log = "Will I dream while I am sleep Dave?"; Server.SelfDestruct = true; }

  • Alan (unregistered) in reply to dkf
    dkf:
    AccessGuru:
    What would Jesus do?
    Keep off-Earth backups, of course. You can't be too careful.

    That would be quite easy to do, all you have to do is transmit your data out into space.

    Recovery would be more difficult as it would involve an FTL drive.

  • (cs) in reply to Mutant
    Mutant:
    My company has suddenly bumped DR right to the top of the priority list after 3 incredible near misses (all unrelated) in 2 weeks.
    Congratulations on making those "near misses" appear to be "unrelated." How did you do it?
  • (cs) in reply to Mutant
    Mutant:
    Lars Vargas:
    And yeah, what about those off-site backups?

    They may have had offsite backups... but without a server to restore them to, it's still going to be a lot of work...

    It's also really hard for people to see ahead to what effects a real disaster will bring before it happens. Especially those signing the bills. Who's really going to predict a server room blowing up?

    My company has suddenly bumped DR right to the top of the priority list after 3 incredible near misses (all unrelated) in 2 weeks. Before then enhancing our current (fairly reasonable) strategy wasn't deemed cost effective. Probably fair enough... I mean, how far do you take it? Nuclear War? Asteriod Strike? The Second Coming?

    Oh, come now. The Second Coming isn't about DR -- it's a huge business opportunity.

    Just imagine all those Scientologists and al Qaeda folk lined up to log in to www.dont-be-a-goat.com.

    The bad news is, most of the money will go to a bunch of raving nutters deep in the Appalachians.

  • Pat (unregistered) in reply to akatherder
    akatherder:
    Why were there a dozen acetylene tanks in a server room? Do they keep some dynamite in the phone closet?

    There weren't They were four concrete rooms away.

  • (cs) in reply to Mutant
    Mutant:
    Lars Vargas:
    And yeah, what about those off-site backups?

    They may have had offsite backups... but without a server to restore them to, it's still going to be a lot of work...

    It's also really hard for people to see ahead to what effects a real disaster will bring before it happens. Especially those signing the bills. Who's really going to predict a server room blowing up?

    My company has suddenly bumped DR right to the top of the priority list after 3 incredible near misses (all unrelated) in 2 weeks. Before then enhancing our current (fairly reasonable) strategy wasn't deemed cost effective. Probably fair enough... I mean, how far do you take it? Nuclear War? Asteriod Strike? The Second Coming?

    Someone suggested this in another (unrelated) article, but it's apropos...

    1. Buy enough deck planking to make a sufficiently large wooden table, and assemble
    2. Buy enough arts-n-crafts paper in a variety of colors to cover said wooden table
    3. Arrange colored paper on wooden table such that each sheet represents one pixel of the appropriate rgb value of the image you wish to back up
    4. Await Google imaging satellite to fly over
    5. Reuse paper and deck for subsequent imaging needs
    6. Voila: cost effective offsite backup of even the biggest images!
  • morry (unregistered)

    Are there any more pictures? I can barely make out anything other than the studs in that one.

  • stark (unregistered)

    Haha, similar incident happened 2 times within a 2 year timespan in the beginning of this decade, in Kista, a suburb to Stockholm, Sweden. The "silicon valley" of Sweden.

    Major power outage. Both citizens and big companies like Ericsson, Tele2, IBM where hit. It took them several days (yes days!) to fix it... Power to the whole area was supplied from a huge tunnel with both primary and secondary mains close together.

  • Ben (unregistered) in reply to Hans
    Hans:
    What would Chunk Norris do?
    Chuck Norris would use a dozen acetylene tanks to create an explosion which would rip through four concrete walls and reduce an entire data center to slag and ash. HTH
  • Man 987876980 (unregistered) in reply to Ben
    Ben:
    Hans:
    What would Chunk Norris do?
    Chuck Norris would use a dozen acetylene tanks to create an explosion which would rip through four concrete walls and reduce an entire data center to slag and ash. HTH
    No, not Chuck Norris, he said ChuNk Norris... who I presume is Chuck's obese brother.
  • null (unregistered)

    Q: How much more dire? A: None. None more dire.

  • (cs)

    Reminds me of my coworker's wife. He gave her a spare car key in case she locked herself out. She put it in the trunk.

  • Man 987876980 (unregistered)

    And of course, Chuck Norris wouldn't need tanks of explosive. He'd just do a roundhouse kick to get the same effect.

Leave a comment on “A Bit More Dire”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article