Release Management Done Right

« Return to Article
  • dkf 2011-06-07 10:29
    It strikes me that Release Management (or whatever you want to label it with) is a sub-discipline of Software Engineering; you can't call a piece of software correctly engineered if it is not possible to release it into use. Only once the code is truly released can a feature be truly said to be finished, as customers (wisely) don't want to futz around with build internals.
  • Steve The Cynic 2011-06-07 10:37
    dkf:
    It strikes me that Release Management (or whatever you want to label it with) is a sub-discipline of Software Engineering; you can't call a piece of software correctly engineered if it is not possible to release it into use. Only once the code is truly released can a feature be truly said to be finished, as customers (wisely) don't want to futz around with build internals.

    +1

    Also note that most descriptions of the classical waterfall model of software engineering include "phases" that are after release, and therefore the process of releasing software, by definition, is part of the discipline (ha!) of software engineering.

    Some words I wrote over twenty years ago in a term paper when I was in university, quoted here because they remain as true now as they were then: "Software engineering is hard." Indeed, experience has taught me that these words are a grotesque understatement.
  • Anonymous Cow-Herd 2011-06-07 10:39
    Frist (±2)

    Today's post clearly was not in any way a thinly-veiled commercial for BuildMaster, honest!

    __
    Note from Alex: drat, caught me red-handed! That's right, this whole thing *wasn't* a thinly-veiled opportunity for me to link to Jenga at Amazon. No, no, I don't have a sci-fi bluray habit that's fed by referral links. But seriously, you should check out Jenga, it's a ton of fun for the whole family. And while you're there (visting from this link of course), check out all the other Great Stuff Amazon has to offer! Jenga.
  • Rootbeer 2011-06-07 10:47

    When do they get to the fireworks factory?

    Or to rephrase, now that you've explained how build management should work, what was the WTF in your colleague's process and how did it manifest?
  • boog 2011-06-07 10:55
    Good article. After working in IT for a number of years, it all seems like it should be common sense.

    Then again, after working in IT for a number of years, it's disturbing how many people still need these concepts repeatedly explained to them.
  • BuildMaster 2011-06-07 10:57
    Instead of a funny story, we get a two-page advertisement for BuildMaster. Yay Tuesday!
  • doctor_of_common_sense 2011-06-07 10:59
    Article in one line: this is the released build of my comment. (Or did I miss anything?)
  • C-Octothorpe 2011-06-07 11:10
    Rootbeer:
    what was the WTF in your colleague's process and how did it manifest?

    I think his soapbox rant was spurred by his colleague’s misuse of the word "release", although my guess is he uses the terms "build" and "release" interchangeably.

    boog:
    Then again, after working in IT for a number of years, it's disturbing how many people still need these concepts repeatedly explained to them.

    This x100...

    I work in a large insurance company, and I still see "promotions" done the "easy" way: Here is a zip with 6 ascx files, 2 dlls and 3 images, and here is a "file list" of where these files go.

    I don't know if it's because I'm a moron but I just naturally feel way more comfortable deploying everything as one deployment "unit" rather than fucking around with 3 files here, 2 files here, oh, and in the web.config on line 43 (at least on my machine it is), change this line...

    This is compounded by the fact that we have a separate group who handles promoting code between envs. who are NOT developers and have almost no domain knowledge.
  • wheaties 2011-06-07 11:16
    Anonymous Cow-Herd:
    Frist (±2)

    Today's post clearly was not in any way a thinly-veiled commercial for BuildMaster, honest!



    My turn next! Feature our product next! I can write a really great WTF for yas...
  • frits 2011-06-07 11:17
    This sounds like some of the most important parts of a well-controlled configuration management program.
  • anonymous_coward 2011-06-07 11:25
    I work in a large insurance company, and I still see "promotions" done the "easy" way: Here is a zip with 6 ascx files, 2 dlls and 3 images, and here is a "file list" of where these files go.
    <snip>
    we have a separate group who handles promoting code between envs. who are NOT developers and have almost no domain knowledge.


    And this is why you need a file list... to tell the non-developer moving the files what to do. This is common. Keep the change as small as possible (minimum number of files) and provide install directions (you can now audit the actual changes against the provided instructions).

    The easy way would be to zip the entire filesystem image and dump it onto the production server. You just have to hope someone didn't modify something unrelated in dev or intentionally slip in malware. E.g. When you see the change list at an atomic file level, common sense says changes to the insurance prenmium rate file shouldn't also include a change to the login security library.
  • eVil 2011-06-07 11:25
    Lack of funny left me disappointed.
    Also, reliving my own terrible experiences of build management makes me want to cry.

    The only positive thing I've come away with from this, is the inexplicable urge to purchase software from Inedo. Anyone else notice that?
  • Alex Papadimoulis 2011-06-07 11:26
    Rootbeer:

    When do they get to the fireworks factory?

    Or to rephrase, now that you've explained how build management should work, what was the WTF in your colleague's process and how did it manifest?


    At around 2,500 words, I figured it was long enough already... but, it basicaly was what C-Octothorpe said:

    C-Octothorpe:
    I still see "promotions" done the "easy" way: Here is a zip with 6 ascx files, 2 dlls and 3 images, and here is a "file list" of where these files go.


    The added WTF was that there was some bastardized source control branching strategy they used to "promote" these "packages". When something didn't work (read: all the time), they'd add or remove packages of code to try to fix it. Once it worked in QA, all the packages were moved en mass to production. That also never worked... especially for database changes.
  • StupidTheKid 2011-06-07 11:37
    Process WTF are as much related to software as hot gums are related to bus seats. Processes determine how software is designed, implemented, tested and released. Process WTF will very often result in Software WTF. Here is an example from a project I had to review post-mortem:

    1) Process determine that the application will be broken down into components.
    2) Process shows that components are developed separately.
    3) Process has no defined integration activity.
    4) Project is delayed two weeks (from a 12 week schedule) because components are unable to communicate with each others.

    While some people will argue that process themselves are useless, they help a lot when trying to find the problem. In this particular case, it was easy enough to find and to fix. And with this simple improvement, more accurate estimates will be provided from now on, which means happier developer, and less WTF.
  • C-Octothorpe 2011-06-07 11:44
    anonymous_coward:
    And this is why you need a file list... to tell the non-developer moving the files what to do. This is common. Keep the change as small as possible (minimum number of files) and provide install directions (you can now audit the actual changes against the provided instructions).

    The easy way would be to zip the entire filesystem image and dump it onto the production server. You just have to hope someone didn't modify something unrelated in dev or intentionally slip in malware. E.g. When you see the change list at an atomic file level, common sense says changes to the insurance prenmium rate file shouldn't also include a change to the login security library.


    But you're contradicting yourself here: include a file list of where the files should go, then you say to include a zip of the root folder and dump the whole thing (which is what I would do).

    The big problem with the first deployment method is this: we're humans and we screw up... If you include everything, there is no room for error; less moving parts and less chances for us monkeys to futz things up.

    This is compounded once you start including more monkeys ("the separation of duty" deployment guys) in the production line who are told to perform quite a complex set of instructions who know nothing about your application nor the framework it was developed on.
  • C-Octothorpe 2011-06-07 11:48
    Alex Papadimoulis:
    The added WTF was that there was some bastardized source control branching strategy they used to "promote" these "packages". When something didn't work (read: all the time), they'd add or remove packages of code to try to fix it. Once it worked in QA, all the packages were moved en mass to production. That also never worked... especially for database changes.


    Sorry, and *how* long, roughly, did this go on for?
  • Brian White 2011-06-07 11:52
    StupidTheKid:
    Process WTF are as much related to software as hot gums are related to bus seats.


    I'm stumped. What are hot gums and what do they have to do with bus seats?
  • Alex Papadimoulis 2011-06-07 11:59
    C-Octothorpe:
    Alex Papadimoulis:
    The added WTF was that there was some bastardized source control branching strategy they used to "promote" these "packages". When something didn't work (read: all the time), they'd add or remove packages of code to try to fix it. Once it worked in QA, all the packages were moved en mass to production. That also never worked... especially for database changes.


    Sorry, and *how* long, roughly, did this go on for?


    The deployments themselves would last hours (or, sometiems a weekend). Like all insane processes, this one evolved over a decade and existed in this form for quite a many years. They're in the process of fixing things, but it's a long journey.

    Now that i think about it, there is one TDWTF-worthy WTF -- as part of the process, they will "diff" compiled (.NET) assemblies before deploying just to be extra sure. Every now and then, they catch code that was accidently compiled in.
  • C-Octothorpe 2011-06-07 12:04
    Alex Papadimoulis:
    they will "diff" compiled (.NET) assemblies before deploying just to be extra sure. Every now and then, they catch code that was accidently compiled in.


    I just died a little inside.
  • Machtyn 2011-06-07 12:13
    Perhaps I've been lucky. In the two companies I've worked, this was essentially the process. The developers compiled the code and notified QA that the files were ready for testing. The QA testers grabbed the entire project (all of the released files were versioned the same, whether or not they had changed) and tested them.

    For my part, as a QA tester that likes to do a little coding, I banged out a little script that allowed the QA team to snag the latest version, or to grab a previous version in the case that field support is needed. In either case, all files for a specific version were copied.

    Essentially, the script made it easier than navigating out to the shared network drive, copying the files, navigating to the install directory, deleting the existing files, then pasting the copied files. The script was select application - enter version number - do the above steps in script.
  • Paul 2011-06-07 12:25
    Thank you for a great article! Can you explain something about the spreadsheet you linked to? The "Overview" tab shows this:


    Integration 2.1 5 5/17/2010
    Testing 2.1 4 5/16/2010
    Staging 2.0A 1 5/1/2011
    Production 2.0A 1 5/2/2011


    How can Integration ever contain a build that has both a later version than Production and an earlier promotion date? Are the "2010"s supposed to be "2011"s?


  • Mason Wheeler 2011-06-07 12:30
    anonymous_coward:
    I work in a large insurance company, and I still see "promotions" done the "easy" way: Here is a zip with 6 ascx files, 2 dlls and 3 images, and here is a "file list" of where these files go.
    <snip>
    we have a separate group who handles promoting code between envs. who are NOT developers and have almost no domain knowledge.


    And this is why you need a file list... to tell the non-developer moving the files what to do.

    ...or you could just do what civilized developers do when they need to deploy code to non-developers, and create an installer. InnoSetup, for example, is free and makes it trivial to create an installer for a program. (Or a build, for that matter)
  • chron3 2011-06-07 12:32
    Machtyn:

    For my part, as a QA tester that likes to do a little coding...


    Patience my pets, patience... soon he will tumble down that slippery slope, and become one with the dark side...
  • boog 2011-06-07 12:33
    Alex Papadimoulis:
    When something didn't work (read: all the time), they'd add or remove packages of code to try to fix it. Once it worked in QA, all the packages were moved en mass to production.
    Ehh, wha? So you're saying they

    1) randomly flipped switches until they got a version that worked,
    2) gave QA that version,
    3) got QA's official blessing on paper (or whatever), and finally
    4) deployed the version they really wanted that wasn't properly tested (or more accurately was tested and failed, hence QA's refusal to sign off on it)?

    That's pretty sick, but I'm not surprised; I've had the pleasure of working with people like this in the past too. It taught me that if your coworkers think it's to their own benefit to fool QA just to make changes in production, then "doing Release Management right" will be the least of your worries.
  • Alex Papadimoulis 2011-06-07 12:34
    Paul:
    Can you explain something about the spreadsheet you linked to? The "Overview" tab shows this:


    Integration 2.1 5 5/17/2010
    Testing 2.1 4 5/16/2010
    Staging 2.0A 1 5/1/2011
    Production 2.0A 1 5/2/2011


    How can Integration ever contain a build that has both a later version than Production and an earlier promotion date? Are the "2010"s supposed to be "2011"s?


    Whoops - a typo. The "2010"s should be "2011"s. I'll fix and re-upload.
  • Alex Papadimoulis 2011-06-07 12:41
    boog:
    Ehh, wha? So you're saying they

    1) randomly flipped switches until they got a version that worked,
    2) gave QA that version,
    3) got QA's official blessing on paper (or whatever), and finally
    4) deployed the version they really wanted that wasn't properly tested (or more accurately was tested and failed, hence QA's refusal to sign off on it)?


    Almost. You forgot a critical step.

    5) Pray that it actually works

    What was strange about this... no one (mangement, net ops, devs, etc) actually liked the process, they just didn't know anything better. And, they were too clouded by the specifics of their situation to generalize the problem.

    There's no book that presents the concepts in a "down-to-earth" manner like this (that I know of, at least - everything I've seen is more idealistic than realistic), and it's not very sexy to talk about on the blogs (unlike TDD, BDD, RDD, FAD, QDD, MAD, ADD, ETC).
  • Brian White 2011-06-07 12:45
    Machtyn:
    Perhaps I've been lucky. In the two companies I've worked, this was essentially the process. The developers compiled the code and notified QA that the files were ready for testing. The QA testers grabbed the entire project (all of the released files were versioned the same, whether or not they had changed) and tested them.

    For my part, as a QA tester that likes to do a little coding, I banged out a little script that allowed the QA team to snag the latest version, or to grab a previous version in the case that field support is needed. In either case, all files for a specific version were copied.

    Essentially, the script made it easier than navigating out to the shared network drive, copying the files, navigating to the install directory, deleting the existing files, then pasting the copied files. The script was select application - enter version number - do the above steps in script.


    My company even has a homegrown product called buildmaster :)
  • dgvid 2011-06-07 12:55
    Brian White:
    StupidTheKid:
    Process WTF are as much related to software as hot gums are related to bus seats.


    I'm stumped. What are hot gums and what do they have to do with bus seats?


    Hmm, well, after I've walked my dog on a warm day I suspect he has hot gums because he pants with his mouth so wide open that his gums are exposed to the cooling air. And he is a greyhound, so there's a sort-of bus company connection there. But, no, that just doesn't quite work. Never mind. I guess I can't get a bus seat out of my dog's mouth.
  • boog 2011-06-07 13:00
    Alex Papadimoulis:
    Almost. You forgot a critical step.

    5) Pray that it actually works

    What was strange about this... no one (mangement, net ops, devs, etc) actually liked the process, they just didn't know anything better. And, they were too clouded by the specifics of their situation to generalize the problem.
    So it wasn't intentionally deceitful. I guess that's better. I've actually had colleagues in previous jobs that used various tricks to get around whatever "Release Management" processes were in place.

    Alex Papadimoulis:
    There's no book that presents the concepts in a "down-to-earth" manner like this (that I know of, at least - everything I've seen is more idealistic than realistic), and it's not very sexy to talk about on the blogs (unlike TDD, BDD, RDD, FAD, QDD, MAD, ADD, ETC).
    Perhaps if it had a fancy name with initials, more people would find the subject interesting?

    RMDD: Release Management Driven Development.

    Wow, I'm already hooked.
  • some guy 2011-06-07 13:06
    Anonymous Cow-Herd:
    Frist (±2)

    Today's post clearly was not in any way a thinly-veiled commercial for BuildMaster, honest!

    __
    Note from Alex: drat, caught me red-handed! That's right, this whole thing *wasn't* a thinly-veiled opportunity for me to link to Jenga at Amazon. No, no, I don't have a sci-fi bluray habit that's fed by referral links. But seriously, you should check out Jenga, it's a ton of fun for the whole family. And while you're there (visting from this link of course), check out all the other Great Stuff Amazon has to offer! Jenga.


    Hate to be this guy, but Alex you literally made me laugh out loud with that reply.

    Look, I'm about as cynical as it gets, but this article is hardly spamtastic. Of course he's going to mention the product he works on. But it's hardly the typical "advertisement disguised as a technical article" that plagues magazines and the web alike.

    Why do I say that? Because Alex's first suggestion is not BuildMaster, it's a spreadsheet. The product is mentioned by name only three times in the entire article. And the article does not try to claim that BuildMaster and only BuildMaster solves these problems and here is why it's unique. Quite the opposite; what commercial have you seen that states you can use other competing solutions because "It really doesn't take much"?

    Alex wrote an article on something he knows. He knows about it because he works on it.

  • David V. Corbin 2011-06-07 13:34
    Alex, for once, I agree with the vast majority of one of your SoapBox posts.. One quibble:

    "and no two versions can be the same"

    I thing you meant the opposite, in that there can never be two different "things" that are the same version.

    Although extremely rare, you can have the followig scenario:

    a) Create version 1.0 - it shipps and works well.
    b) Create version 2.0 - has changes, but alwo problems
    c) Create version 3.0 BY ROLLING BACK ALL OF THE 2.0 Changes.

    As I said, rare, but this is often prefferable than having to deal with "the older version is the better version" scenarios (especially in marketing).
  • C-Octothorpe 2011-06-07 14:01
    boog:
    RMDD: Release Management Driven Development.


    How do I subscribe to your newsletter/religion?
  • anonymous_coward 2011-06-07 14:06
    C-Octothorpe:
    anonymous_coward:
    And this is why you need a file list... to tell the non-developer moving the files what to do. This is common. Keep the change as small as possible (minimum number of files) and provide install directions (you can now audit the actual changes against the provided instructions).

    The easy way would be to zip the entire filesystem image and dump it onto the production server. You just have to hope someone didn't modify something unrelated in dev or intentionally slip in malware. E.g. When you see the change list at an atomic file level, common sense says changes to the insurance prenmium rate file shouldn't also include a change to the login security library.


    But you're contradicting yourself here: include a file list of where the files should go, then you say to include a zip of the root folder and dump the whole thing (which is what I would do).

    The big problem with the first deployment method is this: we're humans and we screw up... If you include everything, there is no room for error; less moving parts and less chances for us monkeys to futz things up.

    This is compounded once you start including more monkeys ("the separation of duty" deployment guys) in the production line who are told to perform quite a complex set of instructions who know nothing about your application nor the framework it was developed on.


    No disagreement here. Moving files individually is definitely more work, but doing it the easy way has security implications.

    And to the guy who said to make an installer, that may work for desktop software or websites. Once you get into the legacy enterprise systems (think TPF assembler and AS400/mainframes) that tends to fall apart fairly quickly.

  • Brian White 2011-06-07 14:28
    anonymous_coward:
    C-Octothorpe:
    anonymous_coward:
    And this is why you need a file list... to tell the non-developer moving the files what to do. This is common. Keep the change as small as possible (minimum number of files) and provide install directions (you can now audit the actual changes against the provided instructions).

    The easy way would be to zip the entire filesystem image and dump it onto the production server. You just have to hope someone didn't modify something unrelated in dev or intentionally slip in malware. E.g. When you see the change list at an atomic file level, common sense says changes to the insurance prenmium rate file shouldn't also include a change to the login security library.


    But you're contradicting yourself here: include a file list of where the files should go, then you say to include a zip of the root folder and dump the whole thing (which is what I would do).

    The big problem with the first deployment method is this: we're humans and we screw up... If you include everything, there is no room for error; less moving parts and less chances for us monkeys to futz things up.

    This is compounded once you start including more monkeys ("the separation of duty" deployment guys) in the production line who are told to perform quite a complex set of instructions who know nothing about your application nor the framework it was developed on.


    No disagreement here. Moving files individually is definitely more work, but doing it the easy way has security implications.

    And to the guy who said to make an installer, that may work for desktop software or websites. Once you get into the legacy enterprise systems (think TPF assembler and AS400/mainframes) that tends to fall apart fairly quickly.



    Moving files individually is not only more work, it is orders of magnitude more likely to introduce bugs. You can always watch your version control system checkins to catch unrelated files being committed, like with your example of the login security library. It is much better to monitor CODE CHANGES for security issues than to use an error prone release system to accomplish the same thing. That's awfully late in the process to be looking for this, and is probably not the right group of people. TortoiseSVN + Jira + pre-commit hooks to prevent any commit not tied to a Jira number works beautifully for this purpose, you can have devs, qa, etc. see the file list show up in the Jira itself.
  • C-Octothorpe 2011-06-07 14:32
    anonymous_coward:
    No disagreement here. Moving files individually is definitely more work, but doing it the easy way has security implications.


    I agree to a limited degree: what security implications? Do you mean swapping good assemblies with malicious assemblies? I know in .Net at least, you can strongly name your assembly. I guess he can also change the public token and version in the config file to match his self-signed key.

    But then again, a logical extension of that argument is that you shouldn't trust your DBAs because they can smoke production in a flash, or your CEO can be purposely steering your company into the ground to allow a competitor to flourish.

    </meandering, pointless rant>
  • The great Nick 2011-06-07 14:32
    That is why I automated everything. Moving to dev to intr to prod takes 9 clicks.

    And no don't tell me that your situation is special and it cannot be automated. No matter what you think it can.
  • C-Octothorpe 2011-06-07 14:35
    The great Alex:
    That is why I automated everything. Moving to dev to intr to prod takes 9 clicks.

    And no don't tell me that your situation is special and it cannot be automated. No matter what you think it can.


    Brought to you by BuildMaster!
  • Chelloveck 2011-06-07 14:40
    C-Octothorpe:
    Alex Papadimoulis:
    they will "diff" compiled (.NET) assemblies before deploying just to be extra sure. Every now and then, they catch code that was accidently compiled in.


    I just died a little inside.


    I've seen similar things. At a place I worked, firmware testing was a long and arduous process. We'd finished the whole test process and were ready to release when we realized we hadn't incremented the version number. Well, okay, we'll just bump it, recompile, and verify that the new image is binary-identical to the old one, except for the one byte where we changed a "1" to a "2".

    We did, and it wasn't. And there was much wailing and gnashing of teeth.

    As was the custom of the time, our version number was stored as a pair of MAJOR, MINOR strings. The old version was 1.1, or "1" and "1". The new one was to be 1.2; "1" and "2". Now, guess what the C compiler likes to do with two identical const strings? It optimizes one of them away, of course. So when we went to 1.2 we went from a single string to two distinct strings, which changed the offset of *everything* farther up in memory. Which was damn near everything, period.

    We ended up bumping the version from 1.1 to 2.2, and everyone lived happily ever after. The End.
  • Alex Papadimoulis 2011-06-07 14:49
    David V. Corbin:
    "and no two versions can be the same"

    I thing you meant the opposite, in that there can never be two different "things" that are the same version.


    Keep in mind that the term "version" is really just an alias for a specific build, and all of the same rules apply (immutable, etc). From a release management standpoint, it doesn't matter the build contains, more that it's a wholly deployable component.

    David V. Corbin:
    Although extremely rare, you can have the followig scenario:

    a) Create version 1.0 - it shipps and works well.
    b) Create version 2.0 - has changes, but alwo problems
    c) Create version 3.0 BY ROLLING BACK ALL OF THE 2.0 Changes.


    It's actually not that uncommon.

    In this case, even if the files within two different builds (say 1.0 and 3.0) are identical down to the byte, the builds are axiomatically different:
    * different build numbers
    * different created dates
    * different promotions
    * etc
  • n 2011-06-07 14:52
    anonymous_coward:

    And to the guy who said to make an installer, that may work for desktop software or websites. Once you get into the legacy enterprise systems (think TPF assembler and AS400/mainframes) that tends to fall apart fairly quickly.


    Your saying that these OSs do not have a built-in scripting language that the operator is using to manually install the packages? They don't have an OS package management system that you can create an repository and a package for your software with (a la, apt/dpkg or yum/rpm)?

    Also, whats wrong with a tarball with a script that does the checking and moving of files (which is honestly what an "installer" does anyway)?
  • C-Octothorpe 2011-06-07 15:01
    n:
    Also, whats wrong with a tarball with a script that does the checking and moving of files (which is honestly what an "installer" does anyway)?


    I've actually resorted to this just because the guy doing the deployment, again, has no domain knowledge and screwed things up repeatedly. Essentially all they now have to do is populate a few key/value pairs in a token file (i.e. prod DB password), and the script handles the rest...

    Hmm, I just reinvented the wheel, didn't I?
  • Kiloman 2011-06-07 16:11
    Funny, my insurance company does it the same way. Blues Unite!
  • muffin 2011-06-07 16:13
    We make new releases several times a day, and my boss is happy. Why bother testing it, when we have customers to do it for us. If they don't complain, it means it works, and if it works, it means we got happy customers.
  • Marvin the Martian 2011-06-07 16:53
    As a biologist, I recommend a Capture-Release approach.

    Basically, you release a few versions or instances into the wild, and later you try to recapture them. For software, if the only running ones you come across are not the ones you released, you're in trouble; conversely for animals, if the only ones you can capture are the ones just released then you're also in trouble (as it seems that species is locally not present anymore).
  • ContraCorners 2011-06-07 17:42
    C-Octothorpe:
    boog:
    RMDD: Release Management Driven Development.


    How do I subscribe to your newsletter/religion?

    Simple. Just get promoted to management.
  • boog 2011-06-07 17:57
    ContraCorners:
    C-Octothorpe:
    boog:
    RMDD: Release Management Driven Development.

    How do I subscribe to your newsletter/religion?

    Simple. Just get promoted to management.
    I was just going to say "read the Internet," but yeah, you could also get promoted to management.

    Those guys get invited to all kinds of parties.
  • Jim 2011-06-07 18:10
    Anonymous Cow-Herd:
    Frist (±2)

    Today's post clearly was not in any way a thinly-veiled commercial for BuildMaster, honest!

    __
    Note from Alex: drat, caught me red-handed! That's right, this whole thing *wasn't* a thinly-veiled opportunity for me to link to Jenga at Amazon. No, no, I don't have a sci-fi bluray habit that's fed by referral links. But seriously, you should check out Jenga, it's a ton of fun for the whole family. And while you're there (visting from this link of course), check out all the other Great Stuff Amazon has to offer! Jenga.


    Does someone have a problem that a site owner advertises their product on their own (FREE) site?

    Oh - How did Alex get past Akismet with all those links...
  • Beryllium 2011-06-07 19:43
    You know, it would go along way to removing the confusion of the build/build orange/orange thing if it was possible to change one of the words.

    build (binaries)/candidate (changeset attempt)
    orange (the colour)/tangerine (the flavour)

    But I'm sure someone has a solid argument against this that I haven't considered (as always seems to be the case when anyone posts their opinion to the Internet :) )
  • B. R. Azil 2011-06-07 20:11
    Alex, minor point, but it's 27b/6 (27b stroke 6), not 27b-6 (27b dash 6).
  • TheCPUWizard 2011-06-07 20:16
    Brian White:
    TortoiseSVN + Jira + pre-commit hooks to prevent any commit not tied to a Jira number works beautifully for this purpose, you can have devs, qa, etc. see the file list show up in the Jira itself.


    Not really, on the "works beautifully" (although it *might* be better than nothing - except for the false sense of security). I have been involved with many a team using this configuration, and have yet to see an SVN+Jira solution that was robust unless the team spent multiples of $100K (in developer salaries) for customization.

    It is far to easy to put the wrong JIRA # in the SVN commit, or to include a file which really belogs to a different change, or a change within a file that does not relate speficially to that JIRA. But people are lulled into a false sense of "security".

    A true ALM environment (every from concept through decomissioning and archiving - including build/relese management)can and should have excellent integration across all elements.
  • Henning Makholm 2011-06-07 20:25
    Alex Papadimoulis:
    Keep in mind that the term "version" is really just an alias for a specific build, and all of the same rules apply (immutable, etc). From a release management standpoint, it doesn't matter the build contains, more that it's a wholly deployable component.

    This is certainly a nice ideal, and does make everybody's life a couple orders of magnitude easier when it can be enforced. But what when it cannot? Scenario:

    Say we have a large, complex production environment running. On day D it all runs a beautiful, immutable, monolithic build where every little bit of code was compiled in one big operation from one precise point in version-control code.

    On day D+1 it's suddenly revealed that the VP of Sales has committed the company to being able to interact with a particular out-of-house web service. And the web service turns out to be incompatible on some deep level with the existing code -- most of the required middleware is already there, but the web service insists, for perverse reasons known only to its developers, that it will sometimes send a "305 Use proxy" response, and the HTTP engine deep down at the bottom of the stack is not prepared to deal with that.

    Now some developer spends a number of days recoding the HTTP engine such that it can react properly to 305. This involves rather a lot of hand-refactoring of code that underlies 75% of everything directly or indirectly, and who knows what little corner case may have accidentally been broken in the process. There is no way a complete rebuild based on that change will end up in production without at least a month of meticulous QA.

    But the salesman promised that we'd have the web service integration up and running by tomorrow. What to do? Take just the server that provides contains the new feature, compile it against the bleeding-edge semi-trusted source tree, and then install a single instance of that in production, where it will interact with the existing, tested servers from the old build. The new server is still high risk, of course, but at least if it fails the failure will be contained; we can configure everything such that existing customers without insane web services will not be handled by that instance at all. Then start working a fresh complete build through QA at a saner pace.

    In that particular case I posit that such a piecewise deployment is the best one can make of a bad situation. However, it does mean that Production now runs a chimera of software, where different components comes from different times of the source control history. It's not a single monolithic build anymore.

    If I understand you correctly, your very vocabulary will refuse to deal with the resulting situation at all, so the process you advocate (whether it is based on whiteboards, or spreadsheets, or specialized tracking software) will be unable to help, not even with managing the task of getting Production back to a purer, less confusing sate.

    (The problem is of course not limited to production environments. For example, if one has the time to spare, one would want to do some integration testing of the chimera configuration before letting loose in production so we'll need to speak about running a chimera in a staging environment).
  • John Shepherd 2011-06-07 21:10
    For f%^*'s sake, shoot the salesman. Now.
  • Alex Papadimoulis 2011-06-07 21:11
    Henning Makholm:
    Alex Papadimoulis:
    Keep in mind that the term "version" is really just an alias for a specific build, and all of the same rules apply (immutable, etc). From a release management standpoint, it doesn't matter the build contains, more that it's a wholly deployable component.

    This is certainly a nice ideal, and does make everybody's life a couple orders of magnitude easier when it can be enforced. But what when it cannot?


    To summarize the scenario you described: a "highly experimental" feature needs to be introduced in production to solve the problem for a known subset of users.

    This is not an uncommon scenario, and is actually a SOP at some places. There are two good ways to address this (isolation by configuration and isolation by instance) and both are compatible with this process.

    The scenario you described is isolation by instance, and requires multiple instances of the software running (load balanced environment, downloaded software, etc). Some instances become "stable", others become "highly experimental"; this yields two, distinct production environments as well. To do this, two versions are created from two different source trees:


    # /branches/highly_experimental/mycodefile.xyz
    GO_CRAZY()

    # /trunk/mycodefile.xyz
    DO_IT_NORMALLY()



    Eventually (after sufficient testing), the changes are merged in and the "highly experimental" instances can be shut down. From a deployment standpoint, builds of "highly experimental" releases can be promoted through a different set of environments ("highly experimental dev", "highly experimental testing", etc) or juts be promoted straight to "highly experimental prod". The latter is obviously reserved for emergencies.

    The other (and slightly easier to manage) isolation looks like this.

    # /trunk/mycodefile.xyz
    IF (GET_SYS_CONFIG("MY_HIGHLY_EXPERIMENTAL_FEATURE_ENABLED")) THEN
    GO_CRAZY()
    ELSE
    DO_IT_NORMALLY()
    END IF



    This allows "highly experimental" features to be enabled/disabled/testing through configuration instead of through a release process. Of course, it also requires a relatively sane code base to pull off, which is why isolation by instance is used just as often.

    Isolation by instance requires some mad release management skillz, but you end up with a "dashboard" that looks like this:

    ENVIRONMENT BLD# DEPLOYED
    Integration v3.8.1 (deployed mm/dd by ...)
    Testing v3.8.1 (deployed mm/dd by ...)
    Staging v3.7.4 (deployed mm/dd by ...)
    Production v3.7.4 (deployed mm/dd by ...)
    HE Testing v4.1.3 (deployed mm/dd by ...)
    HE Production v4.0.9 (deployed mm/dd by ...)
  • da Doctah 2011-06-07 21:56
    anonymous_coward:
    And to the guy who said to make an installer, that may work for desktop software or websites. Once you get into the legacy enterprise systems (think TPF assembler and AS400/mainframes) that tends to fall apart fairly quickly.



    Rolling out to 27 sites, each with its own customized configuration and set of features, we always had an installer to keep contention at bay.

    His name was Doug.
  • migwellian 2011-06-08 03:47
    In your own words, a build might be something that "can’t be built in the first place". Have you considered that the problem might lie in your own confusing nomenclature, rather than just your colleague's insistence in using the more usual definition?
  • minime 2011-06-08 05:11
    We don't have builds, and all those components, is that bad? only one big executable that comes with everything needed compiled in...
  • Rhialto 2011-06-08 05:32
    C-Octothorpe:
    boog:
    RMDD: Release Management Driven Development.


    How do I subscribe to your newsletter/religion?


    Send $10 to the first person on the list, and $5 to the second. THIS IS ENTIRELY LEGAL! Remove the first person, and add yourself to the end of the list. THIS IS ENTIRELY LEGAL!
  • Anonymous Cow-Herd 2011-06-08 05:47
    some guy:
    Look, I'm about as cynical as it gets, but this article is hardly spamtastic. Of course he's going to mention the product he works on. But it's hardly the typical "advertisement disguised as a technical article" that plagues magazines and the web alike.


    Whoooooosh!
  • Level 2 2011-06-08 06:28
    C-Octothorpe:
    anonymous_coward:
    And this is why you need a file list... to tell the non-developer moving the files what to do. This is common. Keep the change as small as possible (minimum number of files) and provide install directions (you can now audit the actual changes against the provided instructions).

    The easy way would be to zip the entire filesystem image and dump it onto the production server. You just have to hope someone didn't modify something unrelated in dev or intentionally slip in malware. E.g. When you see the change list at an atomic file level, common sense says changes to the insurance prenmium rate file shouldn't also include a change to the login security library.


    But you're contradicting yourself here: include a file list of where the files should go, then you say to include a zip of the root folder and dump the whole thing (which is what I would do).

    The big problem with the first deployment method is this: we're humans and we screw up... If you include everything, there is no room for error; less moving parts and less chances for us monkeys to futz things up.

    This is compounded once you start including more monkeys ("the separation of duty" deployment guys) in the production line who are told to perform quite a complex set of instructions who know nothing about your application nor the framework it was developed on.


    Anonymous_coward said that dumping a zipfile of the entire filesystem in production is the easy way, not that it is the correct way.
  • justsomedudette 2011-06-08 07:56
    I work in a small environment where we do everything ourselves, so I'm really having trouble following your stages:
    “integration” - Writing the code?
    “testing” - Unit tests
    “staging” - No idea
    “production” - Making it live

    Can anyone explain this for me please.
  • cappeca 2011-06-08 08:40
    boog:
    Perhaps if it had a fancy name with initials, more people would find the subject interesting?


    And put it in job applications.

    You know what I hate? Going through every acronym in a job application that I'm not familiar with, just to find out it's something I've been doing all my life. Am I too old?
  • frits 2011-06-08 08:44
    If there was an appropriate time for Nagesh to be crowing about CMM Level 5, that time would be now.
  • Rick 2011-06-08 08:45
    Alex Papadimoulis:
    as part of the process, they will "diff" compiled (.NET) assemblies before deploying just to be extra sure. Every now and then, they catch code that was accidently compiled in.


    I just threw up a little in my mouth.
  • Anonymous Cow-Herd 2011-06-08 08:50
    justsomedudette:
    Can anyone explain this for me please.

    Sure.

    Integration: This is where you put everyone's little pieces together, and realize that they don't fit.
    Testing: This is where you prod it and poke it until it falls apart. Then, when it's fallen apart, you realize that it won't fit together again.
    Staging: This is where you make sure the gaffa tape holds, pretend to the customer that it falling apart is a feature if it doesn't.
    Production: The customer gets to keep all the pieces when it breaks.
  • C-Octothorpe 2011-06-08 10:43
    Anonymous Cow-Herd:
    justsomedudette:
    Can anyone explain this for me please.

    Sure.

    Integration: This is where you put everyone's little pieces together, and realize that they don't fit.
    Testing: This is where you prod it and poke it until it falls apart. Then, when it's fallen apart, you realize that it won't fit together again.
    Staging: This is where you make sure the gaffa tape holds, pretend to the customer that it falling apart is a feature if it doesn't.
    Production: The customer gets to keep all the pieces when it breaks.


    You forgot the last step: PROFIT!
  • BentFranklin 2011-06-08 10:44
    Blogging done right
  • Mark 2011-06-08 10:49
    Anonymous Cow-Herd:
    justsomedudette:
    Can anyone explain this for me please.

    Sure.
    <snip whimsical definition>


    Or, mapping concepts to your likely environment

    Integration: Build all projects that make up your product (may be only one). Run the unit tests.
    Testing: Take that build and try it out
    Staging: You likely don't have that.
    Production: Put it on the live server

    Better would be
    Integration: Run an automated build of everything on the build server, including a run of all unit tests.
    Testing: Take that build and do whatever manual testing you want to get in before hitting customers. Run automated tests (that are not unit tests) if you have them.
    Staging: Maybe have an extra server to let select customers try the new version before general deployment.
    Production: Put it on the live server
  • justsomedudette 2011-06-08 10:59
    Thanks Mark I understand now. Anonymous Cow-Herd I kind of got yours, apart from the staging bit, but now it makes more sense. Cheers.
  • boog 2011-06-08 11:34
    cappeca:
    You know what I hate? Going through every acronym in a job application that I'm not familiar with, just to find out it's something I've been doing all my life.
    Indeed; a job app has a minimum requirement of 4 years working with FRESH (Fully-REinvented Software Hodgepodge - which I'll add has only been around for 3 years), but when you look it up you see it's really just an impoverished version of RELIC (RELiable Industry Components), which you've used for more than a decade (or two).

    The part I hate is going in for the job interview. The culture around FRESH has not only reinvented the technology of RELIC, but reinvented the (spoken) language around it as well. So how do you convince these FRESH-fluent interviewers that you understand the concepts, when you don't know how to speak their language?

    I can be a good bullshitter from time to time, but never when it really counts.
  • boog 2011-06-08 11:37
    frits:
    If there was an appropriate time for Nagesh to be crowing about CMM Level 5, that time would be now.
    I haven't seen anything from him in a while (registered version, that is). Maybe he got bored, since we seemed to enjoy his trolling more than he did.
  • Anonymous Cow-Herd 2011-06-08 12:05
    Whimsy aside,
    Mark:
    Integration: Run an automated build of everything on the build server, including a run of all unit tests.
    Testing: Take that build and do whatever manual testing you want to get in before hitting customers. Run automated tests (that are not unit tests) if you have them.
    Staging: Maybe have an extra server to let select customers try the new version before general deployment.
    Production: Put it on the live server

    That's a pretty sensible set of definitions. If you're doing something customer-specific, staging might be where the customer does user acceptance testing before approving it for production.
  • Asiago Chow 2011-06-08 12:06
    Very strange article.

    In my line we are heavily audited/examined (we have Deloitte and various state/federal entities looking through our work item tracking/source control/releases/QA process/etc at least once a year) and need to have an accountable system.

    When a release goes through initial QA and a defect is found, we do another build. A release may have 20 builds or 2000.

    However, once the release makes it past a certain checkpoint, defects found in a release are addressed in subsequent release(s). This is during the final QA and deployment process and may occur before or after a release is deployed to customers. Normally, low impact defects found in this period are not show stoppers and the release proceeds forward. Sometimes, with a significant defect, a release is finished but never deployed and a new (minor or point) release which addresses the defects is created - in which case go back to the top of the previous paragraph and start over.

    So imagine you are having lunch with me. My phone rings. I answer. You hear half the conversation ... “Fine! I guess we’ll just do a new release for QA.” ... Alex's head explodes ....

    Take a chill pill dude.
  • Anonymous Cow-Herd 2011-06-08 12:24
    boog:
    Indeed; a job app has a minimum requirement of 4 years working with FRESH

    Am I the only one that finds temporal requirements nonsense? "Essential: 5 years experience of $language" - does something magical happen on the 5th anniversary that couldn't have happened after, say, 2 years and 324 days? Any reason why people that write these job descriptions seem to think that IT people are singularly incapable of learning new things? I don't recall ever seeing an advert for an accountant that required the candidate to have "3 years experience of writing in two-page A4 ledgers using blue ink". , yet I've heard of a Java shop that turned down several candidates purely because they hadn't used the shop's framework of choice.
  • boog 2011-06-08 12:40
    Anonymous Cow-Herd:
    Am I the only one that finds temporal requirements nonsense? "Essential: 5 years experience of $language" - does something magical happen on the 5th anniversary that couldn't have happened after, say, 2 years and 324 days? Any reason why people that write these job descriptions seem to think that IT people are singularly incapable of learning new things?
    No, I think the real reason for such requirements is to simply assign numbers to the candidate. Screeners/interviewers don't know technology or how to evaluate the quality of a technical candidate. But they do know numerical comparisons. They know the crap out of numerical comparisons.
  • frits 2011-06-08 12:57
    boog:
    Anonymous Cow-Herd:
    Am I the only one that finds temporal requirements nonsense? "Essential: 5 years experience of $language" - does something magical happen on the 5th anniversary that couldn't have happened after, say, 2 years and 324 days? Any reason why people that write these job descriptions seem to think that IT people are singularly incapable of learning new things?
    No, I think the real reason for such requirements is to simply assign numbers to the candidate. Screeners/interviewers don't know technology or how to evaluate the quality of a technical candidate. But they do know numerical comparisons. They know the crap out of numerical comparisons.


    As long as it's apples and apples. Throw them a pear and they'll die of hunger.
  • Vincent 2011-06-08 13:35
    I'd just like to thank you for this article - nothing much else to say, I just wanted to give you some feedback and there is no "like" button ;-)

    I spent countless hours of meetings and more annoying "if you have a minute" meetings interrupting my zone, ending in more confusion on this subject, starting the conversation from scratch over and over again.
  • Mick 2011-06-08 14:21
    I advocate using the ARGH process

    Apply
    Random
    Guesswork
    Hueristics
  • xtremezone 2011-06-09 00:10
    s/shameless/shameful/gi
  • cloudberry 2011-06-09 07:11
    This is an excellent article, explaining in a concise way the basic concepts of release management. With a little tweaking, it could, and maybe should, be an introductory chapter to any explanation of release management.
  • foo 2011-06-09 09:53
    Henning Makholm:

    This is certainly a nice ideal, and does make everybody's life a couple orders of magnitude easier when it can be enforced. But what when it cannot? Scenario:


    You do the experimental work on a Branch in your source control system, and give that customer a release from the Branch.

    Our version numbering system consists of two numbers:
    An external, ever incrementing version that marketing increment as they see politically fit to keep customers happy (e.g. do we go from 3.5 to 3.5.1, 3.6 or 4.0? Depends on how we want customers to perceive the new release).
    An internal version: major.branch.subbranch.build. Uniquely identifies the release in a meaningful way, and we can query source control for the exact version of code files which produced it (first step of the build machine? Add a label to every file with the version of the build we're doing.)
  • Henning Makholm 2011-06-09 14:26
    Alex Papadimoulis:
    The scenario you described is isolation by instance, and requires multiple instances of the software running (load balanced environment, downloaded software, etc). Some instances become "stable", others become "highly experimental"; this yields two, distinct production environments as well. To do this, two versions are created from two different source trees:

    Fair enough. But notice that even though the two production environments may be "distinct" in some appropriate sense, they are not independent. The HE instances need to work together with (exchange data, report billing information, redirect clients to/from, etc) the usual stable instances. Therefore from the point of view of the software that manages the run-time configuration of the entire production environment, there cannot be more than one "production" ensemble, even though it may happen to run a mixture of code from different builds.

    There's nothing inherently wrong in using different terminology between release management and configuration management. But they'd better not be so different that the terminology itself invites confusion ...

    The other (and slightly easier to manage) isolation looks like this.

    # /trunk/mycodefile.xyz
    IF (GET_SYS_CONFIG("MY_HIGHLY_EXPERIMENTAL_FEATURE_ENABLED")) THEN
    GO_CRAZY()
    ELSE
    DO_IT_NORMALLY()
    END IF
    But if the crazy new way requires changes to basic framework/utility code which things that are not supposed to crash, you will need to copy-paste clone all of that code in order to be sure that DO_IT_NORMALLY() indeed does it exactly normally. That results in a source tree with rampant code duplication in it, which entails maintenance problems all of its own.
  • Henning Makholm 2011-06-09 14:36
    foo:
    Henning Makholm:

    This is certainly a nice ideal, and does make everybody's life a couple orders of magnitude easier when it can be enforced. But what when it cannot? Scenario:


    You do the experimental work on a Branch in your source control system, and give that customer a release from the Branch.

    In the scenario I describe, customers are not given releases at all. The software is developed and deployed in-house in order to provide a particular service to customers.

    How and where to branch is a source-control issue, which is distinct from release management (though they obviously inform one another). The point here is that getting the experimental feature to work at all require deep source changes that threaten the stability of the existing non-experimental features. Because everything that serves non-experimental customers has remain stable, there must be a run-time interface between code from the stable build and an experimental build somewhere.
  • Alex Papadimoulis 2011-06-09 16:25
    Henning Makholm:
    even though the two production environments may be "distinct" in some appropriate sense, they are not independent. The HE instances need to work together with (exchange data, report billing information, redirect clients to/from, etc) the usual stable instances. Therefore from the point of view of the software that manages the run-time configuration of the entire production environment, there cannot be more than one "production" ensemble, even though it may happen to run a mixture of code from different builds.


    Good point; from a configuration mangement standpoint, they don't need to be separate environments. I guess in this case, I would set the deployment plan like...


    IF (RELEASE_WORKFLOW = "STANDARD") THEN
    GET_SOURCE_CODE("/TRUNK")
    COMPILE()
    DEPLOY_COMPONENT("CORELIB", "MAIN_SERVERS")
    ELSE IF (RELEASE_WORKFLOW = "HIGHLY_EXPERIMENTAL") THEN
    GET_SOURCE_CODE("/BRANCHES/" + RELEASE_NUMBER)
    COMPILE()
    DEPLOY_COMPONENT("CORELIB", "HIGHLY_EXPERIMENTAL_SERVERS")
    END IF
    DEPLOY_ALL_COMPONENTS_EXCEPT("CORELIB")


    Obviously for later environments, you could simply use artifacts for that build instead of compiling.


    Henning Makholm:
    If the crazy new way requires changes to basic framework/utility code which things that are not supposed to crash, you will need to copy-paste clone all of that code in order to be sure that DO_IT_NORMALLY() indeed does it exactly normally. That results in a source tree with rampant code duplication in it, which entails maintenance problems all of its own.


    Indeed. There are a ton of architectural patterns that can help, but they need to be in place early on. IOW, you shouldn't add a factory pattern for "emergency" isolations - that came come later (and done carefully).
  • xdiv0x 2011-06-14 12:08
    If there is so much confusion between the contextual definitions of an ambiguously defined word, then the best solution isn't to write a lengthy article discussing the difference between the meanings. It is to assign a DIFFERENT word the less common / more formal definition. No confusion thereafter.

    My brain was constantly confused by the article. I liked it, but I had to read it twice. Just because my already trained mind kept assigning the default definition to each and every instance of the word "build" - the common place vernacular one, the (I get the feeling) "wrong" one. But that's just what we've used since we advanced from Qbasic to C, way before high school and college.... that's the way it is for most of us my age...
  • Jon W 2011-06-14 17:19
    Every SVN commit should be a release. You should release 50 times a day! How else can you get immediate feedback from customers on what you're working on?

    Don't believe me? We've done it for years!
    http://engineering.imvu.com/
    If you actually get your ENTIRE code base under test, it works, and it's fantastic!
  • Henning Makholm 2011-06-15 15:48
    xdiv0x:
    My brain was constantly confused by the article. I liked it, but I had to read it twice. Just because my already trained mind kept assigning the default definition to each and every instance of the word "build" - the common place vernacular one, the (I get the feeling) "wrong" one.
    Which default definition do you mean?

    The common-place vernacular sense of "build" has to do with assembling pieces of timber and fired clay into a house. Any other meaning you might want is idiosyncratic jargon, and you're not going to make yourself understood by waving phrases like "the default definition" about without actually specifying which meaning you're going for.
  • jyd 2011-07-07 16:31
    omg, do you work where I work? wait no, we aren't fixing our deployment process.
  • Manuel 2011-07-26 00:02
    I suggest you have a look at the book on Continuous Delivery:

    http://continuousdelivery.com/

    the build pipeline pretty much addresses most of the points you talk about by introducing visibility in the release process and tying all these aspects together: building, testing, deploying.
  • starsfan56 2012-04-26 22:30
    this is one of the best articles i've read in a long time, very well written and full of meat, not just all talk. thanks!
  • Srikanth T 2014-07-12 01:39
    I like this article, any thoughts on using tools such as Plutora for release management? http://www.plutora.com/release-management-software/