Olympic Build and Packaging Pipelines

When setting up an automated continuous delivery pipeline for our current project, we decided to use RPMs and Yum – the native packaging and software updates platform of our staging, UAT and production environments – instead of more Ruby-esque solutions like Capistrano.

There were several reasons behind that, but by far the most important was the affinity we noticed the operations staff already had with RPMs and Yum: all of the system packages were being taken care of using it, with some deal of auditing thrown in: has this file changed since we installed the package? Why?

Making a decent RPM out of a Rails (or Sinatra) project wasn’t very hard: a bit of head-scratching and a few passes through Maximum RPM later, we had something we could work with.

The main headaches were working out which packages were necessary to build the RPM (the BuildRequires part), figuring out how to reliably install all of the gem dependencies from bundler into the packaged RPM (bundle install --deployment helped), and which changes needed to be made to the application to rely solely on environment variables set by /etc/default/[app]. This way, we wouldn’t have any configuration that could vary across environments coming from the package itself.

The next step was setting up a marathon of tests for those RPMs. They already contained unit tested code, and we built more obstacles: functional and integration tests, performance micro-benchmarks, metrics analysis and some manual inspection in UAT. With each step weeding out bad candidates, only truly excellent builds can get to production.

To be able to easily visualize how excellent our builds were, we came up with a simple and effective naming scheme, in time for the Olympics: precious metals.

A unit tested RPM would start out in the “tin” repository. Another step in the pipeline gets triggered and deploys it to a smoke test machine. If that works, it gets promoted to the “bronze” repository. Functional and integration tests cause it to be promoted to “silver”, and so on through “gold” (ready to go) and “platinum” (in production already).

To the operations staff, this makes a lot of sense: the production machines are fetch updates from “gold”, UAT environments look at “silver” and so on. It’s trivial to configure Yum to do that – chances are you alraedy did it when setting up your distribution –, and its output is very easy to read and understand when something goes wrong.

Looking back at the Capistrano days, my only regret is not having done this sooner!