• CCCCC (unregistered)

    I work for a company that is deploying a (long overdue) upgrade to an engineering system used worldwide. Months were spent migrating and converting data to a staging server. The contractors doing grunt work on the project decided to do a DR test. After all, why not, the system wasn't in production yet, so what would be the harm? They did the "D" part quite admirably, wiping out all the data. When it came to the "R" part, they were unable to restore anything. That's when someone noticed that the only copy of the painstakingly converted data was on the staging server that had been wiped. This set our project back at least a month, and I would have loved to have been a fly on the wall for those conversations.

  • Cheese (unregistered)

    Why don't you have naming conventions for servers?

  • (cs) in reply to Steve The Cynic
    Steve The Cynic:
    ochrist:
    I've seen something similar to this, but I'm not sure I get what exactly happened.

    Did they use the staging server instead of a production server? And why did that bring something down? (the boss is claiming that some business critical app is not in production because of this)

    Reading comprehension failure.
    I got 10 paragraphs in and went back to re-read because I thought maybe I'd misunderstood and Victor was in the second phase (deploying patches to prod after 24 hours without problems on staging). His "I haven't promoted anything to production in weeks" (as opposed to "I've never even heard of this app", which is what was previously stated) is a bit confusing.
  • Capt. Obvious (unregistered) in reply to C-Derb
    C-Derb:
    jarfil:
    And what is that preposterous idea of anyone "objecting" before patching/upgrading the staging servers? If stuff breaks, it breaks, that's what they are for.
    I agree that asking permission before rebooting a staging server is overly polite. But I think it is good practice since the warning email is just a courtesy warning that gives developers a heads as to *why* stuff is going to break, and now is a good time to take a break and read TDWTF.com.
    The warning is for the occasional e-mail response that reads something like
    For the love of god don't. There's a problem that I've been debugging all night and I need server X to stay up.
  • Me, over there (unregistered) in reply to Nagesh
    Nagesh:
    Guran:
    Some Damn Yank:
    Guran:
    TRWTF here is production clients accessing a staging server. Few things can cause more badness than staging machines communicating with the outside world.
    True enough, but I've got a real WTF for you: staging machines that run Windows when the production machines run Linux. No shit. And everyone wondered why the web site didn't run right in production after it tested so well in development.

    Happened to me too...

    But the real fun begins when someone does a test run with production data on a not-properly-sandboxed staging server. Especially if your application involves something like mail notification to customers.

    Isn't java running everywhere?

    Yep - write once, debug everywhere.

  • (cs) in reply to anonymous
    anonymous:
    don:
    "What happened to our production server, Pluto? "
    Sorry, he was demoted for negligibility.
    Fail. He was demoted to a staging server.
    Cheese:
    Why don't you have naming conventions for servers?
    I think the complaint is that as you grow you run out of names from said catagory. I always named mine from phonetic alphabets. My webservers started at alpha bravo charlie delta, and my database servers were apple baker chocolate etc... One day I will grow big enough for this to be a problem. I'm actually kinda impatient about it.
  • (cs) in reply to Sobachatina
    Sobachatina:
    I'll take devil's advocate here.

    This business critical app is probably generating actual revenue. R&D was down to the wire and had to get creative. They bypassed the bureaucracy and procedures and got their work done in record time.

    If it was so critical they should have made come the request from high above and not done things from behind the IT's back.

  • Hank (unregistered)

    Another perfect example illustrating why PMs are utterly useless...

  • Ryan (unregistered) in reply to Smug Unix User

    This is, by far, the most accurate and most entertaining comment (that accurately describes my employment situation precisely) that I have ever seen.

    Thank you for this.

  • just me (unregistered) in reply to Some Damn Yank
    Some Damn Yank:
    Guran:
    TRWTF here is production clients accessing a staging server. Few things can cause more badness than staging machines communicating with the outside world.
    True enough, but I've got a real WTF for you: staging machines that run Windows when the production machines run Linux. No shit. And everyone wondered why the web site didn't run right in production after it tested so well in development.

    OMG, you too ? Most of our users are on Linux and Mac and we use mostly Windows/IE . We do a quick sanity check on other browsers/os. All UI is developed on Windows and let me tell you Safari does looks different.

Leave a comment on “Servers Wild”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article