• Simon Jester (unregistered)

    The real WTF was releasing code to production on a Friday...

  • (cs)

    I am continually amazed at how mindless people become when it comes to these things. There's a critical bug that needs to be fixed right now! But we have 3 layers of approval it has to go through before it can go, and one manager is on vacation, and nobody here is intelligent enough to do some critical thinking to fix it, so the bug stays because everyone is too afraid to deviate from the process.

    This is why we need to bring back critical thinking in school. We're becoming a civilization of mindless drones who can't think outside the box.

  • Le Forgeron (unregistered)

    Schools are making what they are ordered to make by the power that be (i.e. Companies with money): nice obedient workers. Do as ordered, however silly, and you will be fine; Be smart and you will be blamed, or fired or both.

    If it is not on the procedure, it cannot be done. We have orders...

  • (cs)

    TRWTF is forgetting who released code.

  • my name is none of your business (unregistered) in reply to Simon Jester

    as a victim of before weekend-releases I can only second that (no pun intended)

  • miko (unregistered)

    Regarding the fact that he typed "like the hacker in a bad movie" - what happened to the president's daughter in that movie?

  • miko (unregistered)

    ...also, isn't friday just another way of saying tuesday anyway?

  • TroelsL (unregistered)

    I think it is pretty obvious here that they need more rigerous prerelease QA. There needs to be a committee that will ensure this will never happen again.

  • tag (unregistered)

    Why didn't they just delete all data from PerfTest? For instance moving it to another server which replaces Perftest? So it would not harm production anymore, and for changing PerfTest the procedure should much easier than for Production.

    Oh, PerfTest was controlled by QA? OK, but at least cutting the connection between Production and PerfTest (real WTF: Why do these servers have a connection anyway?) should be possible for an admin.

  • WorldTrawler (unregistered) in reply to my name is none of your business

    None taken.

  • Mike (unregistered)

    Well this isn't a tech failure. This is a critical management failure.

    A crisis can arise from anywhere, and that doesn't matter. The problem was the corporate paralysis in dealing with it.

    Someone "really" senior needs to be fired

  • Cujo (unregistered)

    BTDT. I worked in an environment where the "Change Control" team had no idea what they were approving but relied on the buddy system. The simplest changes, in this case dropping an index which changed the response time from 20,000 seconds to 1,200 seconds on a batch job is still languishing from 2006. I still get mails about rebuilding the index from my old job.

    OTOH, the most dangerous changes went right through. At one point I had over 16 months of backed up change requests before I gave up and moved elsewhere.

    I have never figured out how people who know nothing about a system, refused to read the documentation the insisted upon and cannot make decisions are the ones who decide what is important. Captcha: consequat <- Should be a real word.

  • Bruce W (unregistered)

    "Blamestorming" -- one of my favorite corporate terms.

    I am forwarding this article to my boss and my team since this happens all too frequently in my mega corporation.

  • Foo (unregistered) in reply to Bruce W
    Bruce W:
    "Blamestorming" -- one of my favorite corporate terms.

    I am forwarding this article to my boss and my team since this happens all too frequently in my mega corporation.

    Unfortunately, places which run like this are never going to accept the criticism of having it pointed out.

    captcha: genitus: The senior manager at John's company was a real genitus.

  • ZoomST (unregistered)

    I wonder why they didn't even try to rollback the changes and apply the changes+patch later. Oh! I see... QA has no budget for approve even the rollback. But the blame should (always) be on the developers. Shouldn't be? It's the easier target. And by the way... why they didn't detected the iBATIS query problem in the development database? Is it a too simpler version? I really don't get it. And I guess I don't want to know the truth...

  • QJo (unregistered)

    TRWTF:

    A horrible dread crept over John’s skin. “I must have left in some flags, or some settings- it’s pulling data from PerfTest..."

    The exact same build that is deployed in Test should be able to be used without changes in Production. If you need to make amendments to the code base before you perform the Production deployment you're doing it wrong.

  • andytech (unregistered) in reply to QJo
    QJo:
    TRWTF:

    The exact same build that is deployed in Test should be able to be used without changes in Production. If you need to make amendments to the code base before you perform the Production deployment you're doing it wrong.

    This.

    Also, production should be isolated from test environments (firewalls, different authentication credentials, etc) so it is simply not possible for them to communicate inadvertently.

  • hombre (unregistered)

    Sadly, no lesson will be learned from this. These people have no jobs (other than overseeing the real working programmers) and taking away their precious process is just unemployment for them.

    If we report that the process failed, we'll just get a more complex one.

  • Ironside (unregistered)

    Why didn't they release the kraken?

  • Nic (unregistered)

    I dont understand. Why not just shut down the PerfTest enviroment so these errors couldnt occur.

  • (cs) in reply to QJo
    QJo:
    TRWTF:

    A horrible dread crept over John’s skin. “I must have left in some flags, or some settings- it’s pulling data from PerfTest..."

    The exact same build that is deployed in Test should be able to be used without changes in Production. If you need to make amendments to the code base before you perform the Production deployment you're doing it wrong.

    Especially with using an ORM (iBatis is mentioned). The database connection should be external somewhere with the same key (I'm a .NET guy so I'm speaking in terms I know) and a different value for dev/test/production so it runs against the correct database without any manual interaction once deployed.

  • (cs) in reply to Cujo

    The trick to getting changes through is to write in the change request “No downtime required”. It doesn't have to be true. It just has to sound non-threatening (and hence easy to evaluate without understanding anything).

    Or at least that's how it works round here…

  • Craig (unregistered) in reply to ObiWayneKenobi
    ObiWayneKenobi:
    This is why we need to bring back critical thinking in school. We're becoming a civilization of mindless drones who can't think outside the box.

    This forum needs a "Like" button.

  • James (unregistered) in reply to Mike
    Mike:
    Well this isn't a tech failure. This is a critical management failure.

    A crisis can arise from anywhere, and that doesn't matter. The problem was the corporate paralysis in dealing with it.

    Someone "really" senior needs to be fired

    Right the quest is just where is the fail coming from. This to me reeks of some finance team taking activity based costing a little to far. I mean why is budget a factor in getting a change request approved to put a fix into the configuration of a production LOB app?

    How long could it possibly take for QA to generate a second opinion as to if this tiny change to a query statement was reasonable?

    If I understand the article correctly it sounds like the thing is still pulling data from a QA or test table. So its not like there is an actual testable change to approve. Someone just needs to drop what they are doing an approve it.

  • (cs) in reply to tag
    tag:
    Why didn't they just delete all data from PerfTest? For instance moving it to another server which replaces Perftest? So it would not harm production anymore, and for changing PerfTest the procedure should much easier than for Production.

    Oh, PerfTest was controlled by QA? OK, but at least cutting the connection between Production and PerfTest (real WTF: Why do these servers have a connection anyway?) should be possible for an admin.

    Seriously. There should have been a free lunch provided for the IT admin to create a mysterious "glitch" that just happened to cut off the test system.

  • (cs) in reply to Craig
    Craig:
    ObiWayneKenobi:
    This is why we need to bring back critical thinking in school. We're becoming a civilization of mindless drones who can't think outside the box.

    This forum needs a "Like" button.

    Critical thinking and government-run education don't go together.

  • Bring Back TopCod3r (unregistered) in reply to ObiWayneKenobi
    ObiWayneKenobi:
    I am continually amazed at how mindless people become when it comes to these things. There's a critical bug that needs to be fixed right now! But we have 3 layers of approval it has to go through before it can go, and one manager is on vacation, and nobody here is intelligent enough to do some critical thinking to fix it, so the bug stays because everyone is too afraid to deviate from the process.

    This is why we need to bring back critical thinking in school. We're becoming a civilization of mindless drones who can't think outside the box.

    It's not a lack of critical thinking, it's more the fact that no-one gets fired for following the process.

  • (cs) in reply to ObiWayneKenobi
    This is why we need to bring back critical thinking in school. We're becoming a civilization of mindless drones who can't think outside the box.

    ^ This.

  • (cs) in reply to Craig
    Craig:
    This forum needs a "Like" button.
    Dislike.
  • chris (unregistered) in reply to Cujo
    Cujo:
    Captcha: consequat <- Should be a real word.
    As in, portmanteau of "consequence" and "splat"?
  • Crabs (unregistered) in reply to Le Forgeron

    I've worked at several large companies.

    Schools are making what they are ordered to make by the power that be (i.e. Companies with money): nice obedient workers. Do as ordered, however silly, and you will be fine; Be smart and you will be blamed, or fired or both.

    If it is not on the procedure, it cannot be done. We have orders...

    If you're really smart, you find the right way to talk to the right people to work around the process and get your fix in. And at that point, you are the hero.

  • bg (unregistered) in reply to Le Forgeron

    Schools are funded by the state, not by businesses.

  • anonim (unregistered) in reply to bg

    And who funds the state?

  • Dave (unregistered) in reply to Craig
    Craig:
    ObiWayneKenobi:
    This is why we need to bring back critical thinking in school. We're becoming a civilization of mindless drones who can't think outside the box.

    This forum needs a "Like" button.

    The irony.

  • Bring Back TopCod3r (unregistered) in reply to Dave
    Dave:
    Craig:
    ObiWayneKenobi:
    This is why we need to bring back critical thinking in school. We're becoming a civilization of mindless drones who can't think outside the box.

    This forum needs a "Like" button.

    The irony.

    Like.

  • (cs) in reply to Dave
    Dave:
    Craig:
    ObiWayneKenobi:
    This is why we need to bring back critical thinking in school. We're becoming a civilization of mindless drones who can't think outside the box.

    This forum needs a "Like" button.

    The irony.

    I've been waiting for him to post a "that's the joke" reply, but perhaps it wasn't after all...

  • Anom nom nom (unregistered)
    1. Releasing a major version to production on a Friday;
    2. Interoperability between testing and production environments;
    3. No process mapped for critical patches.

    TRWTF was people keeping their jobs after this.

  • Steve (unregistered) in reply to Crabs
    Crabs:
    If you're really smart, you find the right way to talk to the right people to work around the process and get your fix in. And at that point, you are the hero.

    Ahh, to the business. As far as your manager is concerned, you are a pain in the butt who doesn't follow the process.

    This entire thing is simply an example of how IT can become unaligned with the business interests of the company.

  • (cs)

    1. "released .. to production" == QA

    2. OP is a limp dick. In a meeting, grab the fire axe and chop the fucking cable that connects testing and production. Mic-drop the axe as a salute to the gods of victory. Stare briefly and stride out, go to roach coach for lunch so you can smoke a cigarette even though you don't smoke normally.

  • (cs)

    Where is Steve and his GAU-8? That thing helps get patches through...

  • Ironside (unregistered)

    "This is why we need to bring back critical thinking in school. We're becoming a civilization of mindless drones who can't think outside the box."

    That's a thought violation! get back in line citizen or you will be reported.

  • Yazeran (unregistered) in reply to chubertdev
    chubertdev:
    Where is Steve and his GAU-8? That thing helps get patches through...

    Or more likely leads to more patches needed (to the hull of whatever the managers are in... )

    Yazeran

    Plan: To go to Mars one day with a hammer

  • David Carter (unregistered)

    TRWTF is that production code was pointing at a test DB, I think. Isn't this what interfaces are for?

  • Svensson (unregistered) in reply to ObiWayneKenobi
    ObiWayneKenobi:
    I am continually amazed at how mindless people become when it comes to these things. There's a critical bug that needs to be fixed right now! But we have 3 layers of approval it has to go through before it can go, and one manager is on vacation, and nobody here is intelligent enough to do some critical thinking to fix it, so the bug stays because everyone is too afraid to deviate from the process.

    They are afraid for a reason, and it has nothing to do with the schools. If your management is going to punish you for not following the procedure, then what exactly is your incentive to not follow the procedure?

    In your example, the problem is not that people follow the procedure. It is that the procedure is buggy. It fails when somebody is on vacation, because nobody ever thought to provide for alternate management approval in that case.

    Don't blame the employees for doing exactly what you wanted them to do.

    (The captcha was "dolor" -- oddly appropriate...)

  • (cs)

    Someone must take the blame. The fracas will continue with fingers pointing in all directions until "the loser" fails to find some other party to blame. That loser is responsible.

    See, corporate environments mostly work, but all of them that I've worked with are insane, at least in some ways or at some times.

    (Note: This "sketch" is mostly made of back of the envelope figures. But there was a real event, that I witnessed one time, that could probably be dropped in here in place of the sketch, without altering much.)

    Suppose Mr. Bigg decides X should be outsourced. The internal team of 10 costs $1 million per year, for 20,000 hours of support. Another $50,000 per year goes into hardware support and software licensing. Horribly, horribly expensive, must be outsourced!

    So a proposal is made to the outsourcing agency, which quotes an average of $3 million per year. ($250,000 each for a production and test region; $350,000 for initial setup; 10 employees at $100/hr for ongoing support, $2 million; and continuing hardware support at $50,000; contracted for 3 years and totaling about $9 million.)

    Okay, that's a no-brainer, right? $1,050,000 internal versus $3 million external. Internal team is better and cheaper!

    No. Mr. Bigg is determined that outsourcing will occur. He's read a magazine that says it's better, so it is.

    So now the negotiations start. The hardware, software, and hardware/software upgrades can't be altered much, so they stay the same. But the initial setup is buried into the support block, and then the employees in that block are cut to just 2 persons. Revised total cost for 3 years: $2.5 million; just $835,000 per year.

    Yes! Outsourcing wins! APPROVED

    But wait: Two employees can't do the work of ten. Oops, you're over your support hours! Don't worry, additional support hours are a mere $150 each. (Extra support hours are treated as overtime: $100 x 1.5)

    Everything is denied or deferred that possibly can be, but there is still a 9,000 hour overage the first year: Extra $1.35 million bill for extra support for the first year...and pretty much every year thereafter; totaling about $2.2 million per year. Just over twice what the internal team and overhead were costing.

    Final result:

    1. Company pays more every year thereafter.
    2. Company gets less adequate service every year thereafter.
    3. People who push yearly hours above 4000 are blamed for causing the overages. ("But the Whatsits department just had to have that new capability. That's what pushed us over.")
    4. Mr. Bigg goes on to greener pa$ture$, with "outsourcing expert" on his resume.

    And that's insane, right? When it happens like this, yes, it is.

    Really, the problem is that the people who should be accountable, aren't. If there is anything that companies and the politicians that run them are good at, it is passing blame when something really bad happens: Dodging accountability.

    If they make good decisions (as they usually do) that isn't an issue; but when they make bad decisions...look out!

    Addendum (2013-04-22 13:57): Forgot an item in the "Final result" list:

    1. Mr. Bigg writes a magazine article about how outsourcing is better.
  • (cs) in reply to miko
    miko:
    Regarding the fact that he typed "like the hacker in a bad movie" - what happened to the president's daughter in that movie?

    i have news. the president's daughter is alive and well. <img src="http://a57.foxnews.com/global.fncstatic.com/static/managed/img/World/660/371/rosichavez_instagram.jpg?ve=1"" border="0" /> here is her latest photo.

  • (cs) in reply to operagost
    operagost:
    tag:
    Why didn't they just delete all data from PerfTest? For instance moving it to another server which replaces Perftest? So it would not harm production anymore, and for changing PerfTest the procedure should much easier than for Production.

    Oh, PerfTest was controlled by QA? OK, but at least cutting the connection between Production and PerfTest (real WTF: Why do these servers have a connection anyway?) should be possible for an admin.

    Seriously. There should have been a free lunch provided for the IT admin to create a mysterious "glitch" that just happened to cut off the test system.

    This entire story is definitely looking made up.

  • ih8u (unregistered) in reply to QJo
    QJo:
    TRWTF:

    A horrible dread crept over John’s skin. “I must have left in some flags, or some settings- it’s pulling data from PerfTest..."

    The exact same build that is deployed in Test should be able to be used without changes in Production. If you need to make amendments to the code base before you perform the Production deployment you're doing it wrong.

    Exactly:

    #ifdef TEST do_stuff(); #else do_somewhat_or_completely_different_stuff(); #endif

    There's the causa (captcha) of your problem.

  • jay (unregistered) in reply to bg
    bg:
    Schools are funded by the state, not by businesses.

    And more importantly, curriculum is set by the state. Yes, big corporations often manipulate the government to get what they want. But I've never heard of school curriculum being high on any mega-corporation's legislative priorities. So instead other interest groups dominate there, like various social activist groups.

    At least here in the US, just look at our school curricula. Is it filled with "corporations are wonderful" propaganda? No, just the opposite: it's mostly trashing big business as evil and teaching students why socialism is a great and worthy cause. The other things that my kids' schools continually teach are: 1. racism is the root of all evil; 2. tobacco is evil; 3. guns are evil; 4. environmentalism is an unqualified good; 5. homosexuality is good, and any questioning or discussion of this point is unacceptable. Math, science, and history come in at about #42, #67, and #83 on their priority list.

  • (cs) in reply to tag
    tag:
    Why didn't they just delete all data from PerfTest? For instance moving it to another server which replaces Perftest? So it would not harm production anymore, and for changing PerfTest the procedure should much easier than for Production.
    That was my first thought.
    tag:
    (real WTF: Why do these servers have a connection anyway?)
    Exactly. For me to reach our production server I have to go out via one IP address, then back in to the production server on it's own IP address. Even though it's sitting in the same rack as all our other servers, they are not on the same switch.

    They are, however, on the same UPS, which is another WTF that bit our butts about a year ago.

Leave a comment on “The Six Million Dollar Patch”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article