- Feature Articles
- CodeSOD
- Error'd
- Forums
-
Other Articles
- Random Article
- Other Series
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
It strikes me that Release Management (or whatever you want to label it with) is a sub-discipline of Software Engineering; you can't call a piece of software correctly engineered if it is not possible to release it into use. Only once the code is truly released can a feature be truly said to be finished, as customers (wisely) don't want to futz around with build internals.
Admin
Also note that most descriptions of the classical waterfall model of software engineering include "phases" that are after release, and therefore the process of releasing software, by definition, is part of the discipline (ha!) of software engineering.
Some words I wrote over twenty years ago in a term paper when I was in university, quoted here because they remain as true now as they were then: "Software engineering is hard." Indeed, experience has taught me that these words are a grotesque understatement.
Admin
Frist (±2)
Today's post clearly was not in any way a thinly-veiled commercial for BuildMaster, honest!
__ Note from Alex: drat, caught me red-handed! That's right, this whole thing wasn't a thinly-veiled opportunity for me to link to Jenga at Amazon. No, no, I don't have a sci-fi bluray habit that's fed by referral links. But seriously, you should check out Jenga, it's a ton of fun for the whole family. And while you're there (visting from this link of course), check out all the other Great Stuff Amazon has to offer! Jenga.
Admin
When do they get to the fireworks factory?
Or to rephrase, now that you've explained how build management should work, what was the WTF in your colleague's process and how did it manifest?
Admin
Good article. After working in IT for a number of years, it all seems like it should be common sense.
Then again, after working in IT for a number of years, it's disturbing how many people still need these concepts repeatedly explained to them.
Admin
Instead of a funny story, we get a two-page advertisement for BuildMaster. Yay Tuesday!
Admin
Article in one line: this is the released build of my comment. (Or did I miss anything?)
Admin
I work in a large insurance company, and I still see "promotions" done the "easy" way: Here is a zip with 6 ascx files, 2 dlls and 3 images, and here is a "file list" of where these files go.
I don't know if it's because I'm a moron but I just naturally feel way more comfortable deploying everything as one deployment "unit" rather than fucking around with 3 files here, 2 files here, oh, and in the web.config on line 43 (at least on my machine it is), change this line...
This is compounded by the fact that we have a separate group who handles promoting code between envs. who are NOT developers and have almost no domain knowledge.
Admin
My turn next! Feature our product next! I can write a really great WTF for yas...
Admin
This sounds like some of the most important parts of a well-controlled configuration management program.
Admin
And this is why you need a file list... to tell the non-developer moving the files what to do. This is common. Keep the change as small as possible (minimum number of files) and provide install directions (you can now audit the actual changes against the provided instructions).
The easy way would be to zip the entire filesystem image and dump it onto the production server. You just have to hope someone didn't modify something unrelated in dev or intentionally slip in malware. E.g. When you see the change list at an atomic file level, common sense says changes to the insurance prenmium rate file shouldn't also include a change to the login security library.
Admin
Lack of funny left me disappointed. Also, reliving my own terrible experiences of build management makes me want to cry.
The only positive thing I've come away with from this, is the inexplicable urge to purchase software from Inedo. Anyone else notice that?
Admin
At around 2,500 words, I figured it was long enough already... but, it basicaly was what C-Octothorpe said:
The added WTF was that there was some bastardized source control branching strategy they used to "promote" these "packages". When something didn't work (read: all the time), they'd add or remove packages of code to try to fix it. Once it worked in QA, all the packages were moved en mass to production. That also never worked... especially for database changes.
Admin
Process WTF are as much related to software as hot gums are related to bus seats. Processes determine how software is designed, implemented, tested and released. Process WTF will very often result in Software WTF. Here is an example from a project I had to review post-mortem:
While some people will argue that process themselves are useless, they help a lot when trying to find the problem. In this particular case, it was easy enough to find and to fix. And with this simple improvement, more accurate estimates will be provided from now on, which means happier developer, and less WTF.
Admin
But you're contradicting yourself here: include a file list of where the files should go, then you say to include a zip of the root folder and dump the whole thing (which is what I would do).
The big problem with the first deployment method is this: we're humans and we screw up... If you include everything, there is no room for error; less moving parts and less chances for us monkeys to futz things up.
This is compounded once you start including more monkeys ("the separation of duty" deployment guys) in the production line who are told to perform quite a complex set of instructions who know nothing about your application nor the framework it was developed on.
Admin
Sorry, and how long, roughly, did this go on for?
Admin
I'm stumped. What are hot gums and what do they have to do with bus seats?
Admin
The deployments themselves would last hours (or, sometiems a weekend). Like all insane processes, this one evolved over a decade and existed in this form for quite a many years. They're in the process of fixing things, but it's a long journey.
Now that i think about it, there is one TDWTF-worthy WTF -- as part of the process, they will "diff" compiled (.NET) assemblies before deploying just to be extra sure. Every now and then, they catch code that was accidently compiled in.
Admin
I just died a little inside.
Admin
Perhaps I've been lucky. In the two companies I've worked, this was essentially the process. The developers compiled the code and notified QA that the files were ready for testing. The QA testers grabbed the entire project (all of the released files were versioned the same, whether or not they had changed) and tested them.
For my part, as a QA tester that likes to do a little coding, I banged out a little script that allowed the QA team to snag the latest version, or to grab a previous version in the case that field support is needed. In either case, all files for a specific version were copied.
Essentially, the script made it easier than navigating out to the shared network drive, copying the files, navigating to the install directory, deleting the existing files, then pasting the copied files. The script was select application - enter version number - do the above steps in script.
Admin
Thank you for a great article! Can you explain something about the spreadsheet you linked to? The "Overview" tab shows this:
How can Integration ever contain a build that has both a later version than Production and an earlier promotion date? Are the "2010"s supposed to be "2011"s?
Admin
Admin
Patience my pets, patience... soon he will tumble down that slippery slope, and become one with the dark side...
Admin
That's pretty sick, but I'm not surprised; I've had the pleasure of working with people like this in the past too. It taught me that if your coworkers think it's to their own benefit to fool QA just to make changes in production, then "doing Release Management right" will be the least of your worries.
Admin
Whoops - a typo. The "2010"s should be "2011"s. I'll fix and re-upload.
Admin
Almost. You forgot a critical step.
What was strange about this... no one (mangement, net ops, devs, etc) actually liked the process, they just didn't know anything better. And, they were too clouded by the specifics of their situation to generalize the problem.
There's no book that presents the concepts in a "down-to-earth" manner like this (that I know of, at least - everything I've seen is more idealistic than realistic), and it's not very sexy to talk about on the blogs (unlike TDD, BDD, RDD, FAD, QDD, MAD, ADD, ETC).
Admin
My company even has a homegrown product called buildmaster :)
Admin
Hmm, well, after I've walked my dog on a warm day I suspect he has hot gums because he pants with his mouth so wide open that his gums are exposed to the cooling air. And he is a greyhound, so there's a sort-of bus company connection there. But, no, that just doesn't quite work. Never mind. I guess I can't get a bus seat out of my dog's mouth.
Admin
RMDD: Release Management Driven Development.
Wow, I'm already hooked.
Admin
Hate to be this guy, but Alex you literally made me laugh out loud with that reply.
Look, I'm about as cynical as it gets, but this article is hardly spamtastic. Of course he's going to mention the product he works on. But it's hardly the typical "advertisement disguised as a technical article" that plagues magazines and the web alike.
Why do I say that? Because Alex's first suggestion is not BuildMaster, it's a spreadsheet. The product is mentioned by name only three times in the entire article. And the article does not try to claim that BuildMaster and only BuildMaster solves these problems and here is why it's unique. Quite the opposite; what commercial have you seen that states you can use other competing solutions because "It really doesn't take much"?
Alex wrote an article on something he knows. He knows about it because he works on it.
Admin
Alex, for once, I agree with the vast majority of one of your SoapBox posts.. One quibble:
"and no two versions can be the same"
I thing you meant the opposite, in that there can never be two different "things" that are the same version.
Although extremely rare, you can have the followig scenario:
a) Create version 1.0 - it shipps and works well. b) Create version 2.0 - has changes, but alwo problems c) Create version 3.0 BY ROLLING BACK ALL OF THE 2.0 Changes.
As I said, rare, but this is often prefferable than having to deal with "the older version is the better version" scenarios (especially in marketing).
Admin
How do I subscribe to your newsletter/religion?
Admin
No disagreement here. Moving files individually is definitely more work, but doing it the easy way has security implications.
And to the guy who said to make an installer, that may work for desktop software or websites. Once you get into the legacy enterprise systems (think TPF assembler and AS400/mainframes) that tends to fall apart fairly quickly.
Admin
Moving files individually is not only more work, it is orders of magnitude more likely to introduce bugs. You can always watch your version control system checkins to catch unrelated files being committed, like with your example of the login security library. It is much better to monitor CODE CHANGES for security issues than to use an error prone release system to accomplish the same thing. That's awfully late in the process to be looking for this, and is probably not the right group of people. TortoiseSVN + Jira + pre-commit hooks to prevent any commit not tied to a Jira number works beautifully for this purpose, you can have devs, qa, etc. see the file list show up in the Jira itself.
Admin
I agree to a limited degree: what security implications? Do you mean swapping good assemblies with malicious assemblies? I know in .Net at least, you can strongly name your assembly. I guess he can also change the public token and version in the config file to match his self-signed key.
But then again, a logical extension of that argument is that you shouldn't trust your DBAs because they can smoke production in a flash, or your CEO can be purposely steering your company into the ground to allow a competitor to flourish.
</meandering, pointless rant>
Admin
That is why I automated everything. Moving to dev to intr to prod takes 9 clicks.
And no don't tell me that your situation is special and it cannot be automated. No matter what you think it can.
Admin
Brought to you by BuildMaster!
Admin
I've seen similar things. At a place I worked, firmware testing was a long and arduous process. We'd finished the whole test process and were ready to release when we realized we hadn't incremented the version number. Well, okay, we'll just bump it, recompile, and verify that the new image is binary-identical to the old one, except for the one byte where we changed a "1" to a "2".
We did, and it wasn't. And there was much wailing and gnashing of teeth.
As was the custom of the time, our version number was stored as a pair of MAJOR, MINOR strings. The old version was 1.1, or "1" and "1". The new one was to be 1.2; "1" and "2". Now, guess what the C compiler likes to do with two identical const strings? It optimizes one of them away, of course. So when we went to 1.2 we went from a single string to two distinct strings, which changed the offset of everything farther up in memory. Which was damn near everything, period.
We ended up bumping the version from 1.1 to 2.2, and everyone lived happily ever after. The End.
Admin
Keep in mind that the term "version" is really just an alias for a specific build, and all of the same rules apply (immutable, etc). From a release management standpoint, it doesn't matter the build contains, more that it's a wholly deployable component.
It's actually not that uncommon.
In this case, even if the files within two different builds (say 1.0 and 3.0) are identical down to the byte, the builds are axiomatically different:
Admin
Your saying that these OSs do not have a built-in scripting language that the operator is using to manually install the packages? They don't have an OS package management system that you can create an repository and a package for your software with (a la, apt/dpkg or yum/rpm)?
Also, whats wrong with a tarball with a script that does the checking and moving of files (which is honestly what an "installer" does anyway)?
Admin
I've actually resorted to this just because the guy doing the deployment, again, has no domain knowledge and screwed things up repeatedly. Essentially all they now have to do is populate a few key/value pairs in a token file (i.e. prod DB password), and the script handles the rest...
Hmm, I just reinvented the wheel, didn't I?
Admin
Funny, my insurance company does it the same way. Blues Unite!
Admin
We make new releases several times a day, and my boss is happy. Why bother testing it, when we have customers to do it for us. If they don't complain, it means it works, and if it works, it means we got happy customers.
Admin
As a biologist, I recommend a Capture-Release approach.
Basically, you release a few versions or instances into the wild, and later you try to recapture them. For software, if the only running ones you come across are not the ones you released, you're in trouble; conversely for animals, if the only ones you can capture are the ones just released then you're also in trouble (as it seems that species is locally not present anymore).
Admin
Admin
Those guys get invited to all kinds of parties.
Admin
Does someone have a problem that a site owner advertises their product on their own (FREE) site?
Oh - How did Alex get past Akismet with all those links...
Admin
You know, it would go along way to removing the confusion of the build/build orange/orange thing if it was possible to change one of the words.
build (binaries)/candidate (changeset attempt) orange (the colour)/tangerine (the flavour)
But I'm sure someone has a solid argument against this that I haven't considered (as always seems to be the case when anyone posts their opinion to the Internet :) )
Admin
Alex, minor point, but it's 27b/6 (27b stroke 6), not 27b-6 (27b dash 6).
Admin
Not really, on the "works beautifully" (although it might be better than nothing - except for the false sense of security). I have been involved with many a team using this configuration, and have yet to see an SVN+Jira solution that was robust unless the team spent multiples of $100K (in developer salaries) for customization.
It is far to easy to put the wrong JIRA # in the SVN commit, or to include a file which really belogs to a different change, or a change within a file that does not relate speficially to that JIRA. But people are lulled into a false sense of "security".
A true ALM environment (every from concept through decomissioning and archiving - including build/relese management)can and should have excellent integration across all elements.