If you've been following The Daily WTF for the past couple years, you may have noticed an ad or two for Inedo or BuildMaster. But what you probably didn't know (at least, according to those I've met in person), is that Inedo is actually my company, and BuildMaster is the product I designed.

I've always been a bit hesitant to talk about these on The Daily WTF. It's not that I think you'd mind, but I've always felt that you come here to read about curious perversions in information technology, not about my professional attempts at improving software development. So thank you in advance for reading this – if you've been reading for some time, you know I don't make a habit of this.

Five Years in the Making

In late 2007, I had this crazy idea for a product that came out of my various experiences at various companies in various industries, so the team and I spec'd it out, named it "BuildMaster", and developed a decent 0.1 version. But instead of going out and telling the world, I hired someone to do all the "marketingy" things. He had a lot of experience in this area, but unfortunately it was from the past decade – and that meant his plan to have an enterprisey, top-down, salesperson-driven approach with no free trial and no public documentation didn't quite work. I talked more about this on Hansleminutes #287, but suffice it to say that we finally got things turned around in 2012.

That's okay, though. A lot of things happened in the past five years: I matured, Inedo matured, and BuildMaster most certainly matured. In fact, I'm finally proud of what our team's done over the years – which is why I haven't really written about it before today. BuildMaster never quite felt "ready" before, especially not for The Daily WTF – the place where some of my best work is featured. You may not realize this, but the stories we write here take quite a lot of time – 5 hours on average – and when they finally are ready, we're quite proud of how they turn out.  I suppose with Inedo and BuildMaster it took 5 years.

DevOps in a Box

It's always been a struggle for me to explain BuildMaster – at least, until the "DevOps" buzzword came around. One of our new (and admittedly cheesy) marketing slogans "DevOps in a Box", and that seems as good a fit as any.

But this whole DevOps thing – this "movement", if you can even call it that – is so ridiculously simple that I can't believe we actually need a word for it. I'm not kidding. It boils down to two common-sense principles.

  1. increased collaboration across teams – not just development and operations, but everyone involved in the software delivery process
  2. automate everything you can – not just builds and deployments, but the process involved in software delivery

The truth is, we do need this word. Having seen how much better things could have been at different organizations – not only the ones I've worked at, but the ones featured on this very website – a buzzword like "DevOps" shines a light on the problems they didn't even realize they had.

After all, it's in an organization's nature to develop silos, and not only will these silos work against the organizational goals to further their own, but they build defensive procedures around themselves to ensure they'll always win the blame game. So when it comes to automation, you need to do more than just make things automatic – this automation needs to facilitate cross-team collaboration as well. That means that neither the side project that a siloed developer created out of frustration (or boredom), nor the bash script that a siloed operations engineer hacked together, nor the Access database maintained by the release engineer can solve this collaborative problem.

As with many things simple, DevOps is not that simple to implement: cross-team automation can't happen without cross-team collaboration, and cross-team collaboration can't happen without cross-team automation. And while it's obvious that dysfunctional silos are the cause of many organizational problems, it's not so obvious that bringing them closer together is the solution.

I noticed this problem in 2007, and BuildMaster – a cross-team automation platform built for the enterprise – was my idea to fix it. At least, that was the plan. BuildMaster turned out a bit differently than I had imagined.

The BuildMaster Way

We obviously use BuildMaster at Inedo, and I can honestly say that BuildMaster has fundamentally changed the way we develop software. I know – that's a pretty big statement to make, and we are obviously our biggest fan – but the thing is, we never designed BuildMaster for ourselves. We're not an enterprise – heck, we don't even have different teams, let alone silos – and the "enterprise" automation that BuildMaster could offer us seemed a bit much.

But we quickly learned that this risk-mitigating, audit-everything process actually adds real value. In fact, the problem was never the process, but the overhead of the process. By reducing this overhead through simple automation, we created a rigorous process involving six environments (Integration, Testing, Staging, Production, Beta, Release) for a  complex application (web app, database, service, agent, etc), and run through the entire thing (from building from source control to manually testing the installers to doing an automated tweet about the new release) in less than an hour.

The quickest way to change software is to just edit the code directly in production, but with BuildMaster this reckless practice is even more unnecessary. For example, when I make the occasional code change to The Daily WTF, I'll deploy to the test site, do a solid 30 seconds worth of testing, then click the "promote build to production" button… all of which is described here. And when I mess up, I just fix it with another quick release or rollback to the previous release.

Sure, all this could be done this with a couple scripts – actually, that's what I used to do – but BuildMaster not only made this easier and more reliable, it also gave us development patterns and practices that only larger, more experienced, and more advanced teams could manage. That's what made the difference for us.

Our BuildMaster users have experienced the same effect, and this is exactly what made me so proud of what the team and I built.

So what's the secret? Instead of just building an automation tool, we modeled the way software is delivered and then built an extensible system around it.

The Software Delivery Process

While the specifics of how software is created constantly changes, the process has remained constant for some fifty years:

  1. Analyze the problem domain, whether it's the evaluation process for prospective Chotchkie's franchisees or the control software on a thermostat
  2. Design how the problem domain will be represented in software, such as the user interface or database model
  3. Implement the design by writing code and building the system
  4. Validate that the software does what it's supposed to through a testing process
  5. Deliver the validated software to its target environment(s)
  6. Repeat as often as needed

This process applies to all software. It is absolute, universal, and thus uselessly nonspecific. As much as I'd like to be able to describe a more specific process, it's simply not feasible. There are far too many types of software, far too many platforms, and far too many other things to even list. But that hasn't stopped others from trying.

Back in the 1990's, there were several attempts to describe the entire software development process and build an automation platform around it. Not surprisingly, many of those attempts ended up like IBM's Rational Unified Process. While I've never used RUP myself, I've met a lot of users who were vocal critics. Actually, they all were. Now that's not to say there are no RUP fans (one of them works hard to ensure there's no criticism on the Wikipedia article), and that's not to say RUP is wrong; it's just that RUP is not for most modern development organizations.

Thus when designing BuildMaster, I knew it'd be a mistake to take on the entire process, and thus I stuck with "software delivery from source control through production." But even then, I needed to find the right balance between uselessly nonspecific and uselessly over-specific. This is (a subset of) the model/process that I came up with; it's a lot, but then again, software delivery is not a trivial process.

  • Environments represent where Application instances live (Dev, Test, Production), but more generally is a stage of testing
    • Servers represent an actual server (which may be in any number of Environments), and is used within a Deployment Plan
  • Applications are the "generally recognized concept" of an application and can vary from a simple command-line tool to a megalith information system; it consists of…
    • Releases are a planned set of changes to an application, as discussed in detail in Release Management Done Right
      • Builds are attempts at meeting those changes, and are comprised of…
        • Promotions indicate that a Build has been found to be suitable for a given Environment; a promotion almost always will result in an…
          • Execution is the deployment of a Build that has been Promoted to an Environment; this almost always happens at promotion time
    • Database Change Scripts are those run-once-and-only-once scripts described in Database Changes Done Right
    • Promotion Workflows describe the sequence of Environments that a Build must pass through, and the approvals (manual and automated) at each stage
    • Configuration Files are the Environment-specific files that describe how an Application is configured (database connection, logging directories, etc); changes to these files are under a release
    • Deployment Plans consist of a set of actions (specific tasks defined in the web UI) and describe how the Application is deployed to a specific Environment

While this BuildMaster model is certainly not universal, these concepts and automation points can describe the way nearly all software is delivered. This is especially important because it allows you to change the way you deliver software without needing to significantly change the tools you use to deliver it. This model is precisely why BuildMaster ended up being so useful for non-enterprisey development.

BuildMaster in Action

It's challenging to concisely demonstrate how BuildMaster works to an audience with mixed skills. A simple demo shown to advanced developers elicits a "I could just script that" response, whereas an advanced demo can result in that whoosssh sound. We did make an introductory video a while back, and although the (ugly) screenshots are outdated, the concepts are basically the same. I'll also be doing a BuildMaster webinar next week, and I'm always happy to talk more about BuildMaster with you one-on-one. Just let me know directly or fill out the demo form on Inedo.com.

But what I'd actually like is for you to try BuildMaster. I know I've been drinking our Kool-Aid, but I strongly believe that BuildMaster can significantly improve the way you deliver software. We put a lot of effort in making BuildMaster intuitive and easy to get started. It even ships with a sample application called Bit Checker, a handy web app that can verify your bits are all 1's or 0's.

There's a learning curve (especially for experienced developers needing to "unlearn" the "just script it" approach), and we're constantly trying to shorten it with our learning center. There are Q&A forums, tutorials, specific implementations, a reference organization (Kramerica), email, and even live chat built right into the software – and I'm in there, helping answer questions as well. If you give BuildMaster more than a cursory glance and don't give up after the first hurdle, you'll find it was well worth your time investment.

BuildMaster is obviously a commercial product, but I really wanted to make it affordable for all teams, and thus we have the free (and limited) Express Edition and the full-featured Enterprise Edition. Like most freemium software, some teams using Express Edition will run into a frustrating limitation (like, needing that 11th application) – but instead of requiring teams to jump into the Enterprise Edition, we have a bunch of really inexpensive BuildMaster Upgrades that Express Edition users can purchase. Going from ten to unlimited applications, for example, is $69. I have absolutely no idea if this is a good way to sell software, but I wish I could buy software this way, so we're trying it out.

The good news is you don't need to worry about our pricing model because I'll just give you BuildMaster Enterprise Edition for free. It's my way of saying thanks for making it all the way through this article (or at least skimming well enough to find the link). All you need to do is just download BuildMaster, send me some feedback, and I'll send you a license.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!