• (cs)

    So the sump dump dumped dump lumps which caused server dumps of dumped core...?

  • Ozz (unregistered)

    One of our newer buildings (about 18 months old) has a switch room with several dozen network switches in it. The Powers That Be didn't see the need for expensive gas fire suppression systems like we have in the other buildings, so these switches sit about 24" under the water sprinkler heads.

  • Robin (unregistered)

    They're not sinks, they're basins. Thank you.

  • (cs)

    this is why I encourage off-site backups

  • bkDJ (unregistered) in reply to chubertdev
    chubertdev:
    this is why I encourage off-site backups
    Whether the sewage gets backed up on-site or off-site, someone somewhere is going to be traumatized and have to wear gloves.
  • Maltz (unregistered) in reply to Maurits
    Maurits:
    I'm glad that "bullet" was put in quotes to make it more clear that he didn't dodge a real bullet but rather a metaphorical bullet.

    Oh, it was real. It just wasn't made of lead...

  • Albert from Ottawa (unregistered) in reply to Cursorkeys
    Cursorkeys:
    We have the main drain-pipe (with a flimsy inspection plate on it) for the factory roof come though the corner of the server room. When we have storms it literally groans under the huge head of pressure.

    A few months ago a different (potable water) pipe burst in the ceiling and rained into the server room. Turns out Dell racks are pretty waterproof from above as we had no real damage.

    I was really surprised that the drain-pipe wasn't the culprit of our near-disaster. Why the hell does it even run inside anyway...

    In Canada these storm water pipes are routed through heated spaces so they work when the outside temperatures are still below or near freezing but there is liquid water on the roof. Think about just after sundown on a warm spring day or (worse) a rain storm on snow on a flat roof.

    Plugged roof drains are a nasty problem that can destroy a roof after only a couple freezing cycles. Then where will your precious servers be? And frozen black water lines are just scary. Think steam clearance Mike Rowe style.

    We have heating cables in the storm drains of open parking garages around here. Many of our apartment parking lots have block heater plugs. Someone else can explain "block heaters" to Floridians and U.S. Georgians.

  • (cs) in reply to Albert from Ottawa
    Albert from Ottawa:
    In Canada these storm water pipes are routed through heated spaces so they work when the outside temperatures are still below or near freezing but there is liquid water on the roof. Think about just after sundown on a warm spring day or (worse) a rain storm on snow on a flat roof.

    Thank you, that makes perfect sense now! Still wish they had put it through the other corner of the building (or not put the servers in that room).

  • Jack (unregistered)

    No one seems to be considering why someone would have "server room duty" for a whole day, to the point where he'd worry that he might get in trouble for being away long enough for a restroom trip.

    This smells (sorry) like one of those places that doesn't do anything proactive, but takes pride instead in how quickly they can respond to alarms. And, since the issues that inevitably pop up can't be solved remotely, we must conclude that the room was chock full of Windows "servers".

    That is, before it became chock full of their equivalent.

    I do recall one former employer with a data center supporting a Fortune 500 financial services company. There was a guy whose full time job was walking from one Windows server to the next and rebooting it. As long as he took down only one at a time, the failover would gracefully handle it. That is, if by "gracefully" you accept that any active sessions would get killed.

    Thanks to third party code full of memory leaks, experience had shown that every server had to be rebooted every 24 hours or Bad Things would happen. But I blame Windows. It's more fun. Besides, Real Computers (TM) are difficult enough to learn that it tends to weed out the "developers" who have never heard of things like proper memory management.

  • George (unregistered) in reply to Albert from Ottawa
    Albert from Ottawa:
    Many of our apartment parking lots have block heater plugs. Someone else can explain "block heaters" to Floridians and U.S. Georgians.
    Fortunately, global warming will soon be here to save the day!

    TRWTF is thinking it is sane to live in places where the water periodically turns solid.

  • sunnyboy (unregistered) in reply to Ralph
    Ralph:
    Finally something worse than JavaScript.

    not my much

  • Al Gore (unregistered) in reply to George
    George:
    Fortunately, global warming will soon be here to save the day!
    Try to keep up. We're not calling it Global Warming any more, due to the inconvenient truth that things don't seem to be much warmer after all, and some places are getting cooler.

    Instead, we are now wringing our hands about Climate Change. That term is so much more versatile...

    Everything Must Remain Exactly The Same.

    If not, it is time for Congress to act, taxes to go up, new programs to spring into immortal life! Departments! Bureaus! Legislation! Regulations! Lawsuits! Yipeee!

    Did you know it has been shown that the tectonic plates are moving? Hawaii's big island is growing by several acres per year, all without Congressional authority!

    Something must be done. This is something. Therefore we must do it.

  • Prof. Foop (unregistered)

    The internet is not a big truck. It's a series of tubes.

  • (cs) in reply to Jack
    Jack:
    No one seems to be considering why someone would have "server room duty" for a whole day, to the point where he'd worry that he might get in trouble for being away long enough for a restroom trip.

    This smells (sorry) like one of those places that doesn't do anything proactive, but takes pride instead in how quickly they can respond to alarms. And, since the issues that inevitably pop up can't be solved remotely, we must conclude that the room was chock full of Windows "servers".

    That is, before it became chock full of their equivalent.

    I do recall one former employer with a data center supporting a Fortune 500 financial services company. There was a guy whose full time job was walking from one Windows server to the next and rebooting it. As long as he took down only one at a time, the failover would gracefully handle it. That is, if by "gracefully" you accept that any active sessions would get killed.

    Thanks to third party code full of memory leaks, experience had shown that every server had to be rebooted every 24 hours or Bad Things would happen. But I blame Windows. It's more fun. Besides, Real Computers (TM) are difficult enough to learn that it tends to weed out the "developers" who have never heard of things like proper memory management.

    FYI, your ivory server towers are made of the same thing as that server room...

  • (cs) in reply to Remy Porter
    Remy Porter:
    I spent some time with an etymology dictionary making sure I could get away with it.
    Don't do that. Your readers aren't going to bother with an etymology dictionary, so if you have to look there, you're not going to get away with it. You've probably already spent more time justifying your decision than it would have taken to find a better word.
  • (cs) in reply to Nagesh
    Nagesh:
    Damien:
    About half way through, I kept reading "server room" as "sewer room" - seemed apt.
    You mean there's a difference? I've always had trouble pronouncing Rs anyway.
    Are you sure you're from India?
  • Albert from Ottawa (unregistered) in reply to George
    George:
    Albert from Ottawa:
    Many of our apartment parking lots have block heater plugs. Someone else can explain "block heaters" to Floridians and U.S. Georgians.
    Fortunately, global warming will soon be here to save the day!

    TRWTF is thinking it is sane to live in places where the water periodically turns solid.

    I'm a sailor. Don't get me started about 'hard water season'. I do want to try ice boating though. Hmmm.

    Some sailors seem to like solid water in their drinks though.

  • (cs) in reply to PedanticCurmudgeon

    I probably could have said the server room had been "basted by human waste", which in retrospect, might have been funnier. Or at least more vile.

  • Larry (unregistered) in reply to Remy Porter
    Remy Porter:
    I probably could have said the server room had been "basted by human waste", which in retrospect, might have been funnier. Or at least more vile.
    Don't expect to please anyone on a site that has nitpicking as its #1 reason for existence.
  • Tony (unregistered) in reply to lesle
    lesle:
    Kidneys are easy to cook: you just boil the piss out of them.

    I guess you could boil the $h!t out of some intestines and have both for dinner!

    Captcha: saluto. If you can stand to eat pi$$ and $h!t parts, I saluto you.

  • Miguel (unregistered)

    If it was here in Brasil, probably the SMS could never be delivered.

  • Mike (unregistered) in reply to Ralph

    Nice planning. Seems to be a general trend often at places I've worked the boss's secretary gets a new big monitor before the programmer, programs get built as side projects and are left to live as shared Access databases on someones desktop, etc etc. No one stops to think: "hmm, if this thing is really this critical shouldn't we be willing to spend some resources on it and do something to protect it?"

    At least in this case they TRIED to put the data centre somewhere that made sense (near utilities, etc).

  • Mike (unregistered) in reply to Jack

    I think a bigger problem is a company that has a failover system installed and kills sessions. Why couldn't they just transfer the sessions/redirect everyone over to the server they weren't trying to reboot at the moment?

    Any system can go down. If your only protection from downtime is "I think my OS is pretty solid and will stay up for years and by gosh I think my devs are better than the other guys" rather than true session preserving failover and proper maintenance it is only a matter of time.

    Make systems easy to redirect so that you can do upgrades while staying up, survive hardware failures, knuckle heads pulling the wrong power cord etc. Relying on the presence of very exceptional people at the OS, dev, and ops layer of an org is very risky if for no other reason that at some point a not so exceptional person might squeak their way into the the org and not quite be able to type 100 wpm of awk/regex without errors for years at a time.

  • (cs) in reply to Cursorkeys
    Cursorkeys:
    Albert from Ottawa:
    In Canada these storm water pipes are routed through heated spaces so they work when the outside temperatures are still below or near freezing but there is liquid water on the roof. Think about just after sundown on a warm spring day or (worse) a rain storm on snow on a flat roof.

    Thank you, that makes perfect sense now! Still wish they had put it through the other corner of the building (or not put the servers in that room).

    Some of us are old enough to have known people in our lives who could remember wondering what kind of weirdo would put a toilet inside the house instead of in a separate building some distance away.

  • PHB (unregistered) in reply to Mike
    Mike:
    at some point a not so exceptional person might squeak their way into the the org and not quite be able to type 100 wpm of awk/regex without errors for years at a time.
    Thanks! You've just inspired a great new interviewing process!
  • (cs)

    But it is not explained why this brought all the servers down. It doesn't sound like anything was damaged.

  • (cs) in reply to Ozz
    Ozz:
    The Powers That Be didn't see the need for expensive gas fire suppression systems like we have in the other buildings, so these switches sit about 24" under the water sprinkler heads.
    I once worked in a warehouse with a re-purposed server room. Cooling was non-existent, water sprinklers were left on the ceiling, and the only way to get equipment in was by climbing a two story high narrow stairway.

    After a few years of replacing overheated hardware the owners finally agreed to put in a purpose built server room with proper cooling and a ground level entrance. Since they hadn't needed to replace any water-damaged servers yet it was decided that overhead sprinklers must still be a good idea, so they were built right in.

    What's that old phrase? "Penny wise and completely clueless"?

  • Alpha (unregistered) in reply to DCRoss
    DCRoss:
    the only way to get equipment in was by climbing a two story high narrow stairway.
    Reminds me of one of my early jobs where we sold computers about the size of refrigerators

    ... from an office on the second floor

    ... of a building with no elevators.

    Yeah, company went broke. Can't imagine why.

  • (cs) in reply to Jack
    Jack:
    No one seems to be considering why someone would have "server room duty" for a whole day, to the point where he'd worry that he might get in trouble for being away long enough for a restroom trip.

    This smells (sorry) like one of those places that doesn't do anything proactive, but takes pride instead in how quickly they can respond to alarms. And, since the issues that inevitably pop up can't be solved remotely, we must conclude that the room was chock full of Windows "servers".

    That is, before it became chock full of their equivalent.

    I do recall one former employer with a data center supporting a Fortune 500 financial services company. There was a guy whose full time job was walking from one Windows server to the next and rebooting it. As long as he took down only one at a time, the failover would gracefully handle it. That is, if by "gracefully" you accept that any active sessions would get killed.

    Thanks to third party code full of memory leaks, experience had shown that every server had to be rebooted every 24 hours or Bad Things would happen. But I blame Windows. It's more fun. Besides, Real Computers (TM) are difficult enough to learn that it tends to weed out the "developers" who have never heard of things like proper memory management.

    I've administered Windows servers for years, and have never encountered an issue requiring a daily reboot.

    You can add RAM and CPUs to Windows servers while they're still fucking running. Can you do that with non-Windows servers? But it's useless because you can't remotely administer a Windows box...

    IHBT

  • (cs) in reply to Remy Porter
    Remy Porter:
    I probably could have said the server room had been "basted by human waste", which in retrospect, might have been funnier. Or at least more vile.
    And it rhymes.
  • (cs) in reply to Remy Porter
    Remy Porter:
    I probably could have said the server room had been "basted by human waste", which in retrospect, might have been funnier. Or at least more vile.

    "The problem in the server room was waste-deep."

  • Mathias (unregistered)

    Crackle is unavailable in my region.

  • Gunslinger (unregistered)

    TRWTF is server room duty (doodie?).

  • TR-Paris (unregistered)

    Here, we can truly say that "the sh*t finally hit the fan...".

    In the proper sense.

  • SomeSignGuy (unregistered) in reply to Ralph
    Ralph:
    Finally something worse than JavaScript.

    Likely some of it started as Java, considering modern coffee consumption...

  • (cs)
    Remy Porter:
    The server room was safely secured against anything generated by Sun.
    FTFY
  • (cs) in reply to Bob
    Bob:
    But... but... did he ever find some place where he could go to the bathroom? Don't leave us in suspense like that!
    He probably just had a quick wazz and a dump in the very server room while he was waiting for professional plumbing services. After all, who would ever know?
  • (cs) in reply to lesle
    lesle:
    Kidneys are easy to cook: you just boil the piss out of them.

    No you don't, you fry them in butter, having sliced them in half lengthways and removed the white bit. Delicious with liver, black pudding, brains and sausage. Mind, I'm not eating eggs, I know which end of the hen they come out of.

  • (cs)

    "The office swung into full meltdown mode, but Florian could only think of one thing: technically he was supposed to be in the server room right now. Was he going to get into trouble?"

    TRWTF are sys-admins. You could have just studied a little harder and you wouldn't have to pinch it off to run across the street because the server room is fucked.

  • joel garry (unregistered)

    I once worked in a large gummint building with several floors of data processing, including massive server rooms in vaults with military guards.

    As part of being environmentally responsible, they replaced all urinals with waterless urinals - a layer of oil floats on top, so it don't smell.

    Of course, this was a WWII (or older?) vintage building, so it had iron sewage lines. The thing about waterless urinals that no one had thought about, was, in normal urinals, the urine gets diluted by water. In waterless urinals, undiluted urine is much more acidic, rapidly eating through metal sewage lines.

    This made the large building uninhabitable.

  • kbiel (unregistered) in reply to ParkinT
    ParkinT:
    I *always* wear rubber gloves when doing server work.

    In order to protect myself from a VIRUS.

    What most people call viruses are actually worms and Trojan horses. Still, gloves are effective when dealing with worms. Trojan horses, on the other hand, call for heavier armament.

  • kbiel (unregistered) in reply to Remy Porter
    Remy Porter:
    I agree, it's rarely used in the context of excretory waste, but it can broadly mean waste (its linguistic roots point towards "things that are removed/cut-off/disposed of"). I spent some time with an etymology dictionary making sure I could get away with it.

    What ever. You're probably one of those people who thinks their own offal smells like roses and can't admit when they're wrong. ;)

    BTW, for all I know, you're offal might smell like roses, but I am told that Tauntaun offal smells worse than the outside.

  • (cs) in reply to Bob
    Bob:
    But... but... did he ever find some place where he could go to the bathroom? Don't leave us in suspense like that!
    At that point I woulda just pissed on the floor.
  • Norman Diamond (unregistered)
    coworker:
    "Toilets are all backed up."
    Good. Next they need to test a restore operation, to make sure the backups will work when the offal thing happens.
    Remy Porter:
    Florian was fairly certain he had forgotten what the sun looked like
    It looked offally similar to the dell and hp.
  • Paul (unregistered)

    Surely the title should be "Full Server Backup"

  • (cs) in reply to George
    George:
    Albert from Ottawa:
    Many of our apartment parking lots have block heater plugs. Someone else can explain "block heaters" to Floridians and U.S. Georgians.
    Fortunately, global warming will soon be here to save the day!

    TRWTF is thinking it is sane to live in places where the water periodically turns solid.

    No, TRWTF is living in places where the water routinely turns solid AND THEN issuing political diatribes about how air conditioning is evil because it contributes to global warming and enabled industrial expansion in the South which votes Republican and Republicans are evil warmongers SO CLEARLY Democrats in northern states are morally superior.

    But it happens.

  • (cs)

    I lost my voice laughing.

  • Mark (unregistered) in reply to Ralph
    Ralph:
    Finally something worse than JavaScript.

    I beg to differ ;)

  • Mike Black (unregistered) in reply to Remy Porter
    Remy Porter:
    Read the HTML comments. ;)

    Wow, a whole new world is opening. Seems like there's another story hidden in it.

  • (cs) in reply to Accalia.de.Elementia
    Accalia.de.Elementia:
    Given the morning reading habits of TDWTF readers, I estimate a 37% probability that you, dear reader, first read this article while in the restroom.

    I further estimate that at least one you who regularly work in a server room will investigate the plumbing of your building to determine the possibility of this happening to you.

    We have business area server rooms located in our office areas; we're on the top floor of the building so I have no worries :) Partly because servers aren't my problem; just the software that runs on them.

    I have no idea where the company wide ones are kept at the new site; at the old site all the servers were kept in what was simply called "the tower". You were probably more at risk from tripping over 100 year's worth of detritus than anything else.

Leave a comment on “Server Dump”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article