• Tuxie (unregistered)

    This could be a tale from almost any IT workplace in the world.

  • (cs)

    I've seen something similar to this, but I'm not sure I get what exactly happened.

    Did they use the staging server instead of a production server? And why did that bring something down? (the boss is claiming that some business critical app is not in production because of this)

  • anon (unregistered)
    Ellis Morning:
    I promised R&D a go-live of April 1st
    I can see why they didn't believe you then.
  • Lars (unregistered)

    I would probably start using that server for any and all early patch testing, making it go down pretty frequently.

  • RFoxmich (unregistered)

    "I'm sorry we decommissioned that server as part of an upgrade to our development/staging environment. Since it was a staging server and nothing critical was supposed to be on it all data and applications deployed to it were lost"

    (CAPTCHA damnum)

  • SirCompo (unregistered)

    It's the gimp in the IT Helpdesk that bypassed the usual process that should be held accountable for this.

  • qbolec (unregistered)

    publicly available beta?

  • (cs)

    I never understand someone who immediately tries to defend the indefensible.

    Fair enough if they admit; "Oh yeah, I kinda took a bit of a shortcut there to meet a deadline, but I know it was shit... I had meant to sort it out properly later and just haven't got round to it... sorry about that", but the ones who fuck up and then completely refuse to acknowledge they fucked up: fire them immediately.

  • (cs)

    So the gist of this is to kill all project managers on sight, yes?

    New PM: Hi, I'm the new Project Mana- IT: EXTERMINATE! EXTERMINATE! New PM: Ahhhhh! Ahhhh! (They close in on him)

  • (cs) in reply to SirCompo
    SirCompo:
    It's the gimp in the IT Helpdesk that bypassed the usual process that should be held accountable for this.

    Eh? Whats it gotta do with the IT Helpdesk guy?

    Someone rang him up, and asked him if he gave them permission to do something. Since it probably wasn't the helpdesk's jurisdiction, why shouldn't the guy there just shrug and say "yeah... I don't have a problem with that mate... knock yourself out."

    If some random cunt phoned me up and asked if I was happy for him to do something that was absolutely nothing to do with me, I'd just say "yes" in a slightly confused tone, then hang up the phone. If it turned out that the guy then murdered a high ranking politician, I'm pretty sure I'd be in the clear.

  • Guran (unregistered)

    TRWTF here is production clients accessing a staging server. Few things can cause more badness than staging machines communicating with the outside world.

  • Smug Unix User (unregistered)

    This is why we can't have nice things. This is also the reason that staging servers get locked down as tightly as production. This is is also the reason that getting an application through the process has more layers than an onion. When people don't follow the rules, more rules typically get added as a knee jerk response to the "how did this happen". It appears the company is about midway in the life cycle.

    Stage one of the life cycle it is the wild west and anything goes anywhere. No sign off, no problem. As the company matures they realize that they need things like change management, source control and other industry standards.

    Standards begin small in the middle stage. Things can get done at this stage but the users who had grown to appreciate the speed at which things were done in the beginning stage lose faith in the system and begin to circumvent it. This triggers the final stage.

    Final stage and nothing can be touched without three signatures. Duties are separated to prevent any one area from having control or authority. Red tape is the final product. At this stage progress grinds to a near standstill.

    Bonus stage. Rise of the contractors. Since nothing can be done in the previous stage contract work begins. Work is farmed out to a company in stage 1 or stage 2. The original host company stays alive until they acquire their contract companies.

  • John (unregistered) in reply to Tuxie
    Tuxie:
    This could be a tale from almost any IT workplace in the world.

    Hah, no, only the few with staging servers.

  • (cs) in reply to ochrist
    ochrist:
    I've seen something similar to this, but I'm not sure I get what exactly happened.

    Did they use the staging server instead of a production server? And why did that bring something down? (the boss is claiming that some business critical app is not in production because of this)

    Reading comprehension failure.

    Victor was upgrading the staging servers. In doing so, he took down a production application that was running on one of the staging servers, not a production server. He didn't know this application was running on the staging server (indeed, it clearly shouldn't have been) because the owner of the application had made an end-run around the production deployment process by going through the IT help desk and nobody there even knew they had to tell anybody about it.

    The most gratifying (for Victor and his boss) response is to simply say that (shrug) it's a staging server, and random applications installed there without our knowledge may be taken down at any time. However, it's not a very team-friendly attitude, and people who live in the real world prefer to merely imagine answering like that.

    One interesting secondary WTF is that the IT help desk has install rights on a staging server.

    Oh, and Victor's group needs an effective way to punish the PM's group for the trouble, because otherwise it will happen again, and again, and again.

  • Androtheos (unregistered) in reply to John

    I have a staging area, it's called my development machine :-)

  • QJo (unregistered)

    Several WTFs but the main one, to my mind, is Victor not being notified (even unofficially) that there is a prod app on Staging.

    Perhaps TRWTF is the PM not realising that doing the above will result in the prod app not being on a stable server.

    Or perhaps TRWTF is the PM having responsibility for this sort of process without understanding the fact that this is a problem in the first place. Or maybe he's had a fight with someone who said "No, you can't do that because blah wibble incomprehensible engineery noise so far over your head you can't even hear it whistle" and won.

    At the very least, the timescale of the project being that critical, the fact the deployment needs to be on Staging would have raised enough red flags that notification of this egregious state of affairs would have filtered through to Victor. "Oh boy, this is Staging - how often is that taken down? Every day at 7 am? Sheesh ..."

  • Roger X (unregistered) in reply to eViLegion

    Fire them? Oh no, no, no. THOSE are the go-getters. The ones who think outside the box and do whatever it takes to accomplish the mission. They're management material.

  • (cs)

    Clearly the project manager made sure every user of TTP knew who to blame if there was a problem; and yet no one leaked what was going on to Victor's team.

    And they tell us large conspiracies can't really happen...

  • Pista (unregistered) in reply to RFoxmich
    RFoxmich:
    "I'm sorry we decommissioned that server as part of an upgrade to our development/staging environment. Since it was a staging server and nothing critical was supposed to be on it all data and applications deployed to it were lost"

    +1

    You told'em, dude!

  • (cs) in reply to Roger X
    Roger X:
    Fire them? Oh no, no, no. THOSE are the go-getters. The ones who think outside the box and do whatever it takes to accomplish the mission. They're *management* material.
    The Dilbert principle at work?
  • anon (unregistered)

    ...The guy in charge of promotions at my new job is named Vic... and let's just say none of that sounds impossible with our setup...

  • G (unregistered)

    Big wtf, but only a small fallout.

    We had a group of people from the business side that also knew a little bit about our code base. You could say they blended the line between IT and Business (though they firmly resided in the Business side with all the power that comes with that).

    Well they developed a fun little application in Python that would run through all the data from sensors scattered throughout the US. This data was then stored historically to later be trended on.

    They needed about 4 terabytes of space for their database, but none of theirs were provisioned for that much. So they poked around until they found a large unsuspecting database, and they used that.

    Needless to say that was one of our testing grounds used by just about every group under the sun. One day one of those groups noticed the database was getting pretty big, and they requested all the data be truncated so they could start a new project using that space.

    Well emails go out, and everyone signs off. Problem is the original group wasn't on the email chain, because no one knew they were using the database.

    Well a couple of truncates later 7 months of production sensor data is gone that can never be recovered (This server only managed to get so much space provisioned for a test stack because it had no backups or redundancies)

  • Sobachatina (unregistered)

    I'll take devil's advocate here.

    This business critical app is probably generating actual revenue. R&D was down to the wire and had to get creative. They bypassed the bureaucracy and procedures and got their work done in record time.

    The loose ends were tied up by IT after the fact.

    Sure it's not ideal. Ideally you'd know about every hiccup months in advance with time to plan for them and get all the necessary approvals. In the real world sometimes you have to get things done creatively.

    The real WTF is the confrontational attitude of the PM. He got it done but he must have known it was bad. I hope his undiplomatic tone is simply embellishment to make a better story.

  • Archdeacon of the Process (unregistered)

    This is how "The Process" gets created...

    The Process be praised! All glory to the Process!

  • Alex (unregistered) in reply to Lars

    That was exactly my thought. Aww, looks like the STAGING server had to go down again. Gee, what a shame.

  • golddog (unregistered)

    Seems to me the real WTF is TTP's PM promising a go-live of April 1st, and not "x nmber of days after Development and UAT are finished."

    When there's a dependency, set expectations based on that dependency.

  • (cs)

    In our client location, we cannot send anything to staging. there are 3 servers named Monika, Phibi and Rachel.

    Monika is development. Phibi is test (UAT) Rachel is production.

    I can only put stuff on Monika.

  • Eric (unregistered) in reply to Nagesh

    Nagesh, your naming standard is bad and you should feel bad. Unless Monika makes great cake, Phibi is stupid and Rachel has great tits. Then you're spot on.

  • (cs) in reply to Eric
    Eric:
    Nagesh, your naming standard is bad and you should feel bad. Unless Monika makes great cake, Phibi is stupid and Rachel has great tits. Then you're spot on.

    what part of client location is not understanding by you? i am not naming the machines. some person with addictive tv disorders is doing that.

  • moving through space (unregistered)

    R&D complaining about lost revenue?

    I thought the whole purpose of R&D was to burn through revenue generated by the business units and replace it with government grant money to help avoid excessive taxes through creative accounting.

  • Sasquatch (unregistered) in reply to Nagesh
    Nagesh:
    In our client location, we cannot send anything to staging. there are 3 servers named Monika, Phibi and Rachel.

    Monika is development. Phibi is test (UAT) Rachel is production.

    I can only put stuff on Monika.

    So someone at your company has the entire DVD box set of the Russian Friends knockoff "Comrades"

  • MiniMax (unregistered)

    The WTF is that there no such thing as development servers, testing servers or staging servers.

    They are all PRODUCTION servers - for different groups.

    Servers being used by developers are production servers - for the developers.

    Servers being used for testing are production servers - for developers (and may QA).

    Servers being used or staging are production servers - for QA.

    If not, then a lot of people are not being productive and should be fired.

  • (cs) in reply to Eric
    Eric:
    Nagesh, your naming standard is bad and you should feel bad. Unless Monika makes great cake, Phibi is stupid and Rachel has great tits. Then you're spot on.
    No, because two out of three are misspelled.
  • MiniMax (unregistered) in reply to Nagesh
    Nagesh:
    In our client location, we cannot send anything to staging. there are 3 servers named Monika, Phibi and Rachel.

    Monika is development. Phibi is test (UAT) Rachel is production.

    I can only put stuff on Monika.

    A sysadmin I had the dubious honor of working with, named all his little utility-scripts with dog names....

  • underling (unregistered) in reply to Steve The Cynic

    Victor's boss is nearly too good to be real...

  • Don (unregistered) in reply to SirCompo
    SirCompo:
    It's the gimp in the IT Helpdesk that bypassed the usual process that should be held accountable for this.
    Absolutely... usually by being promoted to senior management for his "ability to quickly turn around requests"...
  • Bracket (unregistered)

    "TTP" stands for "The TTP Project".

  • Roy (unregistered) in reply to Sobachatina

    There's nothing wrong with bypassing the process to achieve a critical business goal but there's a right way and a wrong way to do so and they chose the wrong way.

    The right way is to push the need to get the application into production ASAP up the chain of command to where someone in Victor's chain of command can decree "this has to go live on April 1st - you are authorised to cut whatever corners you need to from your usual process to achieve this but you should treat this as your top priority". If the PM responsible wasn't able to secure this, it can't have been as business critical as he thought he was.

    Frankly for subverting the process the way he did, the PM should have been fired. And there are plenty of industries where doing it that way could drop the company into a heap of regulatory doo-doo.

  • Ozz (unregistered)

    I knew a SysAdmin who named his servers after supermodels. That resulted in some interesting comments like "Heidi Klum went down on me last night."

  • Spits Coffee Through His Nose (unregistered) in reply to Guran
    Guran:
    TRWTF here is production clients accessing a staging server. Few things can cause more badness than staging machines communicating with the outside world.

    This. You'll inevitably get production tickets against dev versions, people doing production work that gets saved in (ephemeral) dev databases... There's just no way to stay sane in that scenario.

  • MrBester (unregistered)
    "I promised R&D a go-live of April 1st. Development and UAT didn’t finish until March 29th."

    "I am not beholden to your empty promises. BTW, the staging server is going to be taken down for 3 days for maintenance."

    Gotta treat people who think you worse than a commodity with the respect they deserve. It's the only way they'll learn.

  • jarfil (unregistered)

    TRWTF is staging servers not getting wiped regularly.

    "You deployed your app on a staging server? Yeah, well, tough luck, they get wiped every morning."

    Also rebooting of staging servers should be automated and just a click away for any developer, no need to call IT.

    "Your apps keep getting rebooted? On a staging server? Go figure."

    And what is that preposterous idea of anyone "objecting" before patching/upgrading the staging servers? If stuff breaks, it breaks, that's what they are for.

    Captcha: persto. Got problems with your staging server? rm -rf /. Presto.

  • Z (unregistered) in reply to Roy
    Roy:
    There's nothing wrong with bypassing the process to achieve a critical business goal but there's a right way and a wrong way to do so and they chose the wrong way.

    The right way is to push the need to get the application into production ASAP up the chain of command to where someone in Victor's chain of command can decree "this has to go live on April 1st - you are authorised to cut whatever corners you need to from your usual process to achieve this but you should treat this as your top priority". If the PM responsible wasn't able to secure this, it can't have been as business critical as he thought he was.

    Frankly for subverting the process the way he did, the PM should have been fired. And there are plenty of industries where doing it that way could drop the company into a heap of regulatory doo-doo.

    The only 'Business Need' that would be served by subverting the process was that the PM would not have to explain to R&D why TTP did not go live on April 1st as promised, (and most likely would get his bonus for finishing on schedule).

    There is no fool like an April Fool.

  • C-Derb (unregistered) in reply to jarfil
    jarfil:
    And what is that preposterous idea of anyone "objecting" before patching/upgrading the staging servers? If stuff breaks, it breaks, that's what they are for.
    I agree that asking permission before rebooting a staging server is overly polite. But I think it is good practice since the warning email is just a courtesy warning that gives developers a heads as to *why* stuff is going to break, and now is a good time to take a break and read TDWTF.com.
  • Not Necessarily the Nagesh (unregistered) in reply to Spits Coffee Through His Nose
    Spits Coffee Through His Nose:
    There's just no way to stay sane in that scenario.

    Staying sane is not usually one of the requirements that makes the final cut.

  • Not Necessarily the Nagesh (unregistered) in reply to MrBester
    MrBester:
    "I promised R&D a go-live of April 1st. Development and UAT didn’t finish until March 29th."

    "I am not beholden to your empty promises. BTW, the staging server is going to be taken down for 3 days for maintenance."

    Gotta treat people who think you worse than a commodity with the respect they deserve. It's the only way they'll learn.

    The satisfaction you glean from this will last nearly as long as it takes to clean out your desk.

  • (cs)

    Yet another TDWTF about someone not filing the proper paperwork. There's no useful information here, nothing to learn from, just more geeks laughing at management mistakes. I'm getting tired of this. This is not a real WTF.

    Unless the WTF is Victor's boss backing him without question.

  • BillClintonIsTheMan (unregistered) in reply to Nagesh
    Nagesh:
    In our client location, we cannot send anything to staging. there are 3 servers named Monika, Phibi and Rachel.

    Monika is development. Phibi is test (UAT) Rachel is production.

    I can only put stuff on Monika.

    Me too...

  • n_slash_a (unregistered) in reply to MrBester
    MrBester:
    "I promised R&D a go-live of April 1st. Development and UAT didn’t finish until March 29th."

    "I am not beholden to your empty promises. BTW, the staging server is going to be taken down for 3 days for maintenance."

    Gotta treat people who think you worse than a commodity with the respect they deserve. It's the only way they'll learn.

    Yes. However, you need to make sure to come up with better excuses. "This server needs to be taken down for a day to install a service pack." "We need to reconfigure the server to emit tachyons."

  • (cs) in reply to Guran
    Guran:
    TRWTF here is production clients accessing a staging server. Few things can cause more badness than staging machines communicating with the outside world.
    True enough, but I've got a real WTF for you: staging machines that run Windows when the production machines run Linux. No shit. And everyone wondered why the web site didn't run right in production after it tested so well in development.

Leave a comment on “Servers Wild”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article