• SomeCoder (unregistered) in reply to Chad
    Chad:
    Don't laugh. My company is currently that guinea pig. We found a vendor with a software product that MOSTLY meets our needs. However, we're paying them big bucks to make major feature changes. After this is all said and done, they'll integrate it into their general codebase and sell it to everyone else.

    That sounds exactly like one of my former companies except that instead of paying the vendor to make major feature changes, we did it ourselves.

    Yes, we did work for the vendor for free that they can then roll into their product. The catch was that our work was pretty highly specialized so at best, they would have to generalize some of it.

  • (cs) in reply to sas
    sas:
    real_aardvark:
    Grovesy:
    Mr. Eff:
    Grovesy:
    Why deploy? just code directly on the server!

    You must be a Sharepoint developer!

    Saddly not, retail developer... as much as you try and drum into the business 'process' and protecting the thing that makes you a few hundred million a year they still force through 1/2 finished features to 'get it out there and get feedback'...

    Nothing like an out of action site loosing millions by the hour to focus your mind on why ohh why you accepted another contract in this industry.

    You're English, aren't you?
    More likely American. English spelling hasn't degenerated as far as that.

    No, as previously stated I am English, I just can't spell.

  • runamok (unregistered) in reply to Grovesy
    Grovesy:
    Why deploy? just code directly on the server!

    I sthis jeff, my coworker? :-)

  • Anonymous (unregistered) in reply to danixdefcon5

    The software that I currently work on was an identical setup - PHP, no VCS, all changes made on production. Fortunately, the guy behind that setup left soon after I came into the picture, and all of that changed... but there were some pretty nice disasters.

  • Anonymous (unregistered) in reply to oppeto

    Nobody expects a completely incorrect Monty Python reference, either.

  • Leo (unregistered)

    Love it. Reminds me when, working on a project, the Project Manager tells me that he just promised the client that it would be delivered that evening. No time to test, except on my work machine where the current version was, ah, slow to execute, which I pointed out but that didn't seem to bother Project Manager.

    When the msi went to SA for deploy, Project Manager told me that I looked really tired and could go home. I was and did. On getting home, the phone rang, CTO telling me that all the servers had gone down, klang klang emergency, and would I come back to work right away.

    Project Manager still works there, I don't. Guess it pays to be drinking buddies with one of the C-team members.

  • Modo (unregistered) in reply to Reaper

    FTA: "The business didn't have to run its usual expensive testing cycle."

    Yes, because when things look ok in UA/QA/PreProd you still need to test them thoroughly because there might be hidden, subtle bugs, but when everything looks ok in Production, then it must be OK!

  • Top Cod3r (unregistered)

    All you college boys think you know how software is developed. The WTF was his boss was trying to be a red tape task master by forcing him to spend a week testing a his change. My guess is he was afraid of the guy getting too good of a reputation as a fast and good coder so he blamed the production outage on him, and then posted the incident on the daily wtf for good measure. What ever happened to teamwork?

  • jonnyq (unregistered) in reply to taylonr

    Same here.

    I've been at my job for almost two years. (PHP sites)

    Zero source control. You test by making backup copies of the relevant files and then copy over the live files when you're done.

    The thing is, as awful as the system is, it works remarkably well - with only 4 developers, there's little chance of stepping on toes, and what problems arise are usually minor. Code changes can go live within a couple hours or even minutes, but the tradeoff is code that's harder to maintain.

    I'm still pushing for true source control and a development environment. I think a dev server is right around the corner, and getting chunks of code moved into SVN is probably going to be one of my top priorities. But when problems are sparse and it's hard to blame problems on a bad dev environment, it's hard to instill a sense of urgency to switch to a real dev environment.

  • Mate (unregistered) in reply to Grovesy
    Grovesy:
    Why deploy? just code directly on the server!

    It's a sad reality that probably dozens of thousands of PHP asshats do this as a daily routine.

  • Fumikazu Hayami (unregistered) in reply to Man 987876980
    Man 987876980:
    Outlaw Programmer:
    I'm beginning to support MasterPlanSoftware's ban on xkcd references.
    I agree, except you should see this one: http://www.xkcd.com/378/
    "Real Programmers" is good too.
  • PhysicsPhil (unregistered) in reply to danixdefcon5
    danixdefcon5:
    At least it was an accident, I've sometimes seen management actually order devs to skip QA and deploy straight into production!
    Outlaw Programmer:
    I'm beginning to support MasterPlanSoftware's ban on xkcd references.
    No, please ... no. The main articles fortunately aren't his "domain". Anyway, why ban them if we like them? Most xkcd jokes are directly or indirectly related with computer science, or even computing WTF's (like the Debian OpenSSL SNAFU) so it's not like its off-topic.

    I for example just don't find the Monty Python overquoting funny, but I'm not about to propose a Monty Python reference ban either. ;)

    Fair enough, but can we require an ICBM address so that we can LART the posters of excess links?

  • PhysicsPhil (unregistered) in reply to Anonymous
    Anonymous:
    Nobody expects a completely incorrect Monty Python reference, either.
    Nobody expects a different XKCD reference
  • (cs) in reply to Matt C
    Matt C:
    http://www.xkcd.com/378/
    How many Matt C's are there here?
  • (cs) in reply to Rich
    Rich:
    With all these 'real programmers use ...' references, i was reminded of The Story of Mel.
    Real programmers use pointers.
  • katastrofa (unregistered) in reply to real_aardvark
    real_aardvark:

    The sad thing is that I suspect that the Specialist (Third Class) Artillery guy in question was posted to the Gulf just before that Iranian civilian airliner was shot down.

    This was the ship's captain's decision.

  • equex (unregistered)

    what is this 'deploy' and 'qa' that you speak of ?

    i must surely be better to program straight into the production code like a real man.

  • teh_n1gz (unregistered) in reply to Grovesy
    Grovesy:
    Why deploy? just code directly on the server!

    Do not joke, that is how I am forced to work pretty much atm :'(

  • Dwedit (unregistered)

    I guess this means you have to write all documentation as if it were to be read by Amelia Bedelia.

  • BlackStar (unregistered) in reply to snoofle

    That's what our company does. Every single fucking day.

    We also code directly on production.

    And there's absolutely noone uses source control (though I've been pushing for months, demonstrated SVN ("oh, that's useful!"), even got the admin to install a server).

    The management thinks this is cool (low turnaround time!) and it's the perfect source material for WTFs (oops, my trigger accidentally dropped the whole table, help me fix that before the client notices!)

    Bah, the pay's good at least.

  • (cs) in reply to grg
    grg:
    >It makes absolutely no sense to do a simulated bombing with a real warhead or tracking practice with a loaded gun.

    I beg to disagree. Time and time again it's been found that the parts you skip will have some flaw that invalidates the usefulness of the whole test.

    In particular with the bombs you have to use the real thing to check out the interfaces between the bomb pod and the plane's electronics. Sometimes something as subtle as the bomb pod's fuel tank being filled with especially cold fuel could cause weird sensor readings. Also they would not have learned that the fuel could seep into the bomb compartment if the plane made a 6-g turn.

    All of which you can find out without the bomb containing a live warhead.
    snoofle:
    Actually, with a big enough gun with a big enough shell, the weight of the shell in the gun makes a difference w.r.t. targeting/trajectory/recoil.
    Again, there is no need for the shell to be a real one. The military has dummy shells for exactly that reason.

    Anyone doing exercises with weapons armed with live ordnance when the exercise does not actually intend for the weapon to be fired needs to be dismissed (or whatever they call it in the military) for criminal negligence.

  • (cs) in reply to Matt.C
    Matt.C:
    Matt C:
    http://www.xkcd.com/378/
    How many Matt C's are there here?

    Well there seems to be Matt.C and Matt C. So they are not the same

  • (cs) in reply to equex
    equex:
    what is this 'deploy' and 'qa' that you speak of ?

    i must surely be better to program straight into the production code like a real man.

    I call this operation of an open heart (at least as far as databases are concerned)

  • MentalImage (unregistered) in reply to Bobbo
    Bobbo:
    MentalImage:
    I've got a vivid mental image of :

    Mr. Bronson shouting "Daaavvve"! ("Kendallll"!) Closeup of Dave (I've substituted the head of John R. General Manager at StartLogic for Dave, as I don't know what Dave looks like) And then the Grange Hill #duh-dit-dah-down# music as Dave realises what he's done...

    Ahh, the memories! Does Mr Bronson's wig come loose shortly after?

    It does now - thanks for the enhancement :-)

  • MattC (unregistered) in reply to ClaudeSuck.de

    I'm Matt C!

  • Paul (unregistered) in reply to brazzy
    brazzy:
    All of which you can find out without the bomb containing a live warhead.

    Ah, but it still needs to weigh as much as a normal bomb.

    Not many people like having a ton of concrete dropped on their heads/houses/cars from an altitude of a few miles...

    In fact, there was talk a few years ago of using concrete bombs as real ordnance. With todays guidance technology, they can be dropped neatly onto the target without causing significant collateral damage (apart from flying bits of debris from the squished tank/building/bridge/terrorist/whatever)

    Yes, using a nuke, or even a live HE bomb in test runs would be dumb, but using a dummy bomb still doesn't mean you should drop it over Chicago...

  • grg (unregistered) in reply to Bappi

    There's a difference between testing and practicing. In addition, if you're going to test your procedure and systems with a live nuke, I'd rather you didn't do it over Chicago. That's what god gave us New Mexico for.

    That would be nice but they had to be prepared for bad weather over the target, which meant bombing by radar, which made New Mexico a poor target.

  • OrangeSpyderMan (unregistered)

    The real WTF is, of course, that there's no segregation of duties. It's f*ck-ups like this that make it a best practice.

  • P (unregistered) in reply to Jake Vinson
    Jake Vinson:
    Not sure if this has been posted yet: http://www.xkcd.com/378/

    I don't think it has been posted yet Jake. But don't worry, I'll post it now:

    http://www.xkcd.com/378/

  • (cs) in reply to grg
    grg:
    I beg to disagree. Time and time again it's been found that the parts you skip will have some flaw that invalidates the usefulness of the whole test.

    In particular with the bombs you have to use the real thing to check out the interfaces between the bomb pod and the plane's electronics.

    I'm sorry, but you have absolutely no clue whatsoever.

    The Air Force Strategic Air Command NEVER, and I repeat NEVER, used live nukes for practice runs over populated areas. Never. That's what proving grounds and test areas are for; you know, test areas like New Mexico and atolls at sea?

    And I do know what I'm talking about; I served in SAC for 6 years during the 70's and 80's, working in both an Organizational Maintenance Squadron and a Field Maintenance Squadron, on the flightline with B-52 bomber and KC-135 tanker aircraft. I even spend time in the nuclear alert area with the bomber pilots and crews.

    Nice try, but no go. Better luck with your fiction in the future.

  • (cs) in reply to Paul
    Paul:
    brazzy:
    All of which you can find out without the bomb containing a live warhead.

    Ah, but it still needs to weigh as much as a normal bomb.

    Not many people like having a ton of concrete dropped on their heads/houses/cars from an altitude of a few miles...

    In fact, there was talk a few years ago of using concrete bombs as real ordnance. With todays guidance technology, they can be dropped neatly onto the target without causing significant collateral damage (apart from flying bits of debris from the squished tank/building/bridge/terrorist/whatever)

    Yes, using a nuke, or even a live HE bomb in test runs would be dumb, but using a dummy bomb still doesn't mean you should drop it over Chicago...

    TRWTF is that they are using concrete bombs. How can you estimate the amount of damage the bomb will cause without actually using it. Calculations are nothing. As one can see with Hiroshima and Nagasaki. Everybody was quite surprised about the efficiency. Without real testing (developers, think about it) you always guess, but never know. Look at the famous Tesla experiment. He must have said a very loud ooooooops when he learned about the effect of his Dead Ray.

    http://www.wired.com/culture/culturereviews/magazine/16-06/st_tunguska http://www.bibliotecapleyades.net/tesla/esp_tesla_2.htm http://www.world-mysteries.com/sci_tesla1.htm http://www.bibliotecapleyades.net/esp_ciencia_tunguska.htm

    So, he did it right (though he only wanted to send a greeting to Peary approaching North Pole by creating north lights). Theory is good, practice and test is better.

  • (cs) in reply to KenW
    KenW:
    grg:
    I beg to disagree. Time and time again it's been found that the parts you skip will have some flaw that invalidates the usefulness of the whole test.

    In particular with the bombs you have to use the real thing to check out the interfaces between the bomb pod and the plane's electronics.

    I'm sorry, but you have absolutely no clue whatsoever.

    The Air Force Strategic Air Command NEVER, and I repeat NEVER, used live nukes for practice runs over populated areas. Never. That's what proving grounds and test areas are for; you know, test areas like New Mexico and atolls at sea?

    And I do know what I'm talking about; I served in SAC for 6 years during the 70's and 80's, working in both an Organizational Maintenance Squadron and a Field Maintenance Squadron, on the flightline with B-52 bomber and KC-135 tanker aircraft. I even spend time in the nuclear alert area with the bomber pilots and crews.

    Nice try, but no go. Better luck with your fiction in the future.

    If you are a developer, do you develop cancer, now?

  • Colin (unregistered) in reply to Grovesy

    Isn't that how everyone codes? If it compiles, it's production ready!!

  • (cs) in reply to Leo
    Leo:
    Love it. Reminds me when, working on a project, the Project Manager tells me that he just promised the client that it would be delivered that evening. No time to test, except on my work machine where the current version was, ah, slow to execute, which I pointed out but that didn't seem to bother Project Manager.

    When the msi went to SA for deploy, Project Manager told me that I looked really tired and could go home. I was and did. On getting home, the phone rang, CTO telling me that all the servers had gone down, klang klang emergency, and would I come back to work right away.

    Project Manager still works there, I don't. Guess it pays to be drinking buddies with one of the C-team members.

    The point of Project Management is that the process should be repeatable.

    This particular process sounds, alas, eminently repeatable.

  • Jan (unregistered) in reply to Grovesy

    Thats not funny when you have been doing it a couple of months...

  • (cs) in reply to jonnyq
    jonnyq:
    Same here.

    I've been at my job for almost two years. (PHP sites)

    Zero source control. You test by making backup copies of the relevant files and then copy over the live files when you're done.

    The thing is, as awful as the system is, it works remarkably well - with only 4 developers, there's little chance of stepping on toes, and what problems arise are usually minor. Code changes can go live within a couple hours or even minutes, but the tradeoff is code that's harder to maintain.

    I'm still pushing for true source control and a development environment. I think a dev server is right around the corner, and getting chunks of code moved into SVN is probably going to be one of my top priorities. But when problems are sparse and it's hard to blame problems on a bad dev environment, it's hard to instill a sense of urgency to switch to a real dev environment.

    Mention the magic words "Credit Crunch" and get them to read Taleb's The Black Swan.

    No, wait, this is a PHP shop, right?

    Scratch that thing about reading.

  • (cs)

    I've just looked at that deployment "script" again, and I think most of us are missing the point. (I certainly was.)

    That has to be the worst deployment script I've ever seen. It seems to boil down to "Squash my jars with new ones."

    It also lacks steps that a QA might consider pertinent, such as, ooh, I dunno, actual testing.

    It also lacks roll-back instructions.

    Even heavily anonymised (although why one would choose to anonymise abstract instructions is beyond me), this is still steaming crap, isn't it?

  • grg (unregistered) in reply to ClaudeSuck.de

    And I do know what I'm talking about; I served in SAC for 6 years during the 70's and 80's, working in both an Organizational Maintenance Squadron and a Field Maintenance Squadron, on the flightline with B-52 bomber and KC-135 tanker aircraft. I even spend time in the nuclear alert area with the bomber pilots and crews.

    That's swell, but in your position you would not have been privy to what was done in the mid 60's at B-58 bases. The weapons on the B-52's are carried internally, so it's possible for the nukes to be made live in flight by insertion of the inner capsule. And you're right, that is never done on a practice run while over the target, it's usually done on the run-in at low level so they can verify that the guy can do the insertion during maximum jinking and turbulence.

    But in the 60's, many of the planes then in use F-86's, F-4's, F-105's, and B-58's were either single-seaters or had the nukes mounted externally or both, making in-flight insertions impossible. And the planes were always in the air, or parked at the end of the runway or on 15-minute alert, so there was no time for insertion, so the nukes were kept strapped on and configured and everything but PAL'd.

    I agree that was kinda insane, but that's the way things were.

  • (cs) in reply to rleibman
    rleibman:
    Bappi:
    Thou shalt not grind thy beans until thou needst them, and then only in such quantity as is necessary, no more.

    Though shall not ROAST the beans until thou needs them.

    Now there's a great post. I can understand not knowing that "thou" isn't spelled "though" because of the perils of English spelling versus pronunciation, but not when it's right in front of your eyes. <g>

  • (cs)

    Hmm, let me see. In order to be able to do what he has done he would need:

    1. Permission (badge?) to enter the building.
    2. Windows logon for his PC.
    3. Windows permissions set up correctly on his PC.
    4. Windows logon for QA server (if it's the same as in three he must be set up as a QA server user).
    5. Windows permissions set up correctly on QA PC.
    6. Permission to logon to QA DB server.
    7. Database permission to logon to QA DB.
    8. QA DB permission setup correctly.
    9. Windows logon for PROD PC.
    10. Windows permissions set up correctly on PROD PC.
    11. Permission to logon to PROD DB server.
    12. Database permission to logon to PROD DB.
    13. PROD DB permission setup correctly.
    14. Know physical location of QA server (for shutdown/startup).
    15. Have access to QA server.
    16. Know physical location of PROD server.
    17. Have access to PROD server.
    18. Instruction on how to find server version.
    19. Know location of necessary files.
    20. Know the who-is-who (for priorisation)
    21. Find the documentation.
    22. Read the documentation.
    23. (Find out about the company and it's culture)
    24. Did I forget something?

    I have never been in a situation where this was all done in one morning before, say, 9 o'clock (Several hours later, the production server went down. And Dave was still there).

    There are 2 possibilities:

    • this story is made up, or worse
    • the QA lead had given him admin permissions with his (QA lead) own passwords and logons before going on holiday.

    Now, that is what I call a WTF.

    Addendum (2008-06-20 11:25): 23. (Find out about the company and it's culture) This one in brackets. Not really needed. Just, on my first days I arrive around 9 in the morning. Then comes the greeting. Then the showing around (toilets, coffee machine...). Introduction to people. Then you are sat on a desk (physically, because nobody can find a chair). Most often you do not even have a comp yet ("They set it up for tomorrow, or so). You're introduced on what you will do the next few days and the next few weeks. There's only 24 hours in a day, even if it's the first one.

  • (cs) in reply to grg
    grg:
    >And I do know what I'm talking about; I served in SAC for 6 years during the 70's and 80's, working in both an Organizational Maintenance Squadron and a Field Maintenance Squadron, on the flightline with B-52 bomber and KC-135 tanker aircraft. I even spend time in the nuclear alert area with the bomber pilots and crews.

    That's swell, but in your position you would not have been privy to what was done in the mid 60's at B-58 bases. The weapons on the B-52's are carried internally, so it's possible for the nukes to be made live in flight by insertion of the inner capsule. And you're right, that is never done on a practice run while over the target, it's usually done on the run-in at low level so they can verify that the guy can do the insertion during maximum jinking and turbulence.

    But in the 60's, many of the planes then in use F-86's, F-4's, F-105's, and B-58's were either single-seaters or had the nukes mounted externally or both, making in-flight insertions impossible. And the planes were always in the air, or parked at the end of the runway or on 15-minute alert, so there was no time for insertion, so the nukes were kept strapped on and configured and everything but PAL'd.

    I agree that was kinda insane, but that's the way things were.

    You're arguing with KenW?

    Do not argue with KenW.

    KenW will twist your faulty logic and historical examples into pretzels.

    You think that's Bad?

    Wait until he calls upon his friend Chuck Norris.

  • (cs) in reply to real_aardvark
    real_aardvark:
    You're arguing with KenW?

    Do not argue with KenW.

    KenW will twist your faulty logic and historical examples into pretzels.

    And then peek you in the nose.

  • Dylan16807 (unregistered)

    Clearly the documentation was at fault. It should have read like this:

    To deploy server onto QA

    1. Log into QA server
    2. Take down server
    3. Run necessary SQL commands
    4. Copy over new jar files
    5. Bring server back up
    6. break;
  • (cs) in reply to ClaudeSuck.de
    ClaudeSuck.de:
    real_aardvark:
    You're arguing with KenW?

    Do not argue with KenW.

    KenW will twist your faulty logic and historical examples into pretzels.

    And then peek you in the nose.

    Gute Freund, dass wurde aber besser sein als: "Poke you in the nose."

    I like "peek," though.

  • (cs) in reply to grg
    grg:
    That would be nice but they had to be prepared for bad weather over the target, which meant bombing by radar, which made New Mexico a poor target.

    You continue to spew nonsense. You don't have to have any particular type of weather to practice bombing by radar. You simply practice bombing by radar. Actual conditions have no bearing on anything.

    I really wish people who know nothing about a topic would refrain from posting about it.

  • (cs) in reply to ClaudeSuck.de
    ClaudeSuck.de:
    If you are a developer, do you develop cancer, now?

    No.

    If you're Claude, and you Suck, is it just in Germany, or is it everywhere?

  • (cs) in reply to grg
    grg:
    But in the 60's, many of the planes then in use F-86's, F-4's, F-105's, and B-58's were either single-seaters or had the nukes mounted externally or both, making in-flight insertions impossible. And the planes were always in the air, or parked at the end of the runway or on 15-minute alert, so there was no time for insertion, so the nukes were kept strapped on and configured and everything but PAL'd.

    I agree that was kinda insane, but that's the way things were.

    Great. The alert aircraft were kept locked and loaded. But they didn't fly the damned things over populated areas for practice runs using live ordnance, in the 50s, 60s, or any other time in any aircraft.

    Quit trying to make excuses for posting nonsense.

  • grg (unregistered) in reply to KenW

    You simply practice bombing by radar.

    Ahem, one question. Does the New Mexico desert, on radar, look anything like Novosibirsk, Vladivostok, Moscow, Gorky, et. al?

  • thedarkside (unregistered) in reply to KenW
    KenW:
    grg:
    That would be nice but they had to be prepared for bad weather over the target, which meant bombing by radar, which made New Mexico a poor target.

    You continue to spew nonsense. You don't have to have any particular type of weather to practice bombing by radar. You simply practice bombing by radar. Actual conditions have no bearing on anything.

    I really wish people who know nothing about a topic would refrain from posting about it.

    Yeah, but on the other hand, watching them be humilated by your good self is a great spectator sport ;-)

  • (cs) in reply to real_aardvark
    real_aardvark:
    ClaudeSuck.de:
    real_aardvark:
    You're arguing with KenW?

    Do not argue with KenW.

    KenW will twist your faulty logic and historical examples into pretzels.

    And then peek you in the nose.

    Gute Freund, dass wurde aber besser sein als: "Poke you in the nose."

    I like "peek," though.

    I wouldn't want to think about poking in somebodies nose, especially before lunch. I prefer the image of somebody peeking into your nose (or vice versa). Aaah, good ol' BASIC! You could even peek into the memory (brain). Or poke. Then peek.

Leave a comment on “Application Lifecycle Mismanagement”

Log In or post as a guest

Replying to comment #:

« Return to Article