Continuous deployment is very buzzword-y right now. I have some really strong opinions about deployment (just ask James Socol or Erik Kastner, who have heard me ranting about this). Here’s what I think, in a nutshell:
You should build the capability for continuous deployment even if you never intend to do continuous deployment. The machinery is more important than your deployment velocity.
Let me take a step back and talk about deployment maturity.
At the immature end, a developer or team works on a system that has no staging environment. Code goes from the development environment straight to production. (I’m not going to even talk about the situation where the development environment is production. I am fully aware that these still exist, from asking questions to that effect of the audience in conference talks.) I’m also assuming, in this era of github, that everybody is using version control.
(I want to point out that it’s the easy availability of services like github that has enabled even tiny, disorganized teams to use version control. VC is ubiquitous now, and that is a huge blessing.)
This sample scenario is very common: work on dev, make a commit, push it out to prod. Usually, no ops team is involved, or even exists. This can work really well in an early stage company or project, especially if you’re pre-launch.
This team likely has no automation, and a variable number of tests (0+). Even if they have tests, they may have no coverage numbers and no continuous integration.
When you hear book authors, conference speakers or tech bloggers talk about the wonders of continuous deployment, this scenario is not what they are describing.
The machinery of continuous deployment
Recently in the Mozilla webdev team, we’ve had a bunch of conversations about CD. When we talked about what was needed to do this, I had a revelation.
Although we were choosing not to do CD on my team, we had in place every requirement that was needed:
- Continuous integration with build-on-commit
- Tests with good coverage, and a good feel for the holes in our coverage
- A staging environment that reflects production – our stage environment is a scale model of production, with the same ratios between boxes
- Managed configuration
- Scripted deployment to a large number of machines
I realized then that the machinery for continuous deployment is different from the deployment velocity that you choose for your team. If we need to, we can make a commit and push it out inside of a couple of minutes, without breaking a sweat.
Why we don’t do continuous deployment on Socorro
We choose not to, except in urgent situations, for a few reasons:
- We like to performance test our stuff, and we haven’t yet automated that
- We like to have a human QA team test in addition to automated tests
- We like to version our code and do “proper” releases because it’s open source and other people use our packages
- A commit to one component of our system is often related to other commits in other components, which make more sense to ship as a bundle
Our process looks like this:
- The “dev” environment runs trunk, and the “stage” environment runs a release branch.
- On commit, Jenkins builds packages and deploys them to the appropriate environment.
- To deploy to prod, we run a script that pulls the specified package from Jenkins and pushes it out to our production environment. We also tag the revision that was packaged for others to use, and for future reference. Note we are pushing the same package to prod that we pushed to stage, and stage reflects production.
- If we need to change configuration for a release, we change it first in Puppet on staging, and then update production the same way.
We do intend to increase our deployment velocity in the coming months, but for us that’s not about the machinery of deployment, it’s about increasing our delivery velocity.
Delivery velocity is a different problem, which I’m wrestling with right now. We have a small team, and the work we’re trying to do tends not to come in small chunks but big ones, like a new report, or a system-wide change to the way we aggregate data (the thing we’re working on at the moment). It’s not that changes sit in trunk, waiting for a release. It’s more that it takes us a while to get something to a deployable stage. That is, deployment for us is not the blocker to getting features to our users faster.
It’s the same old theme you’ve seen ten times before on this blog: everybody’s environment is different, and continuous deployment may not be for everybody. On the other hand, the machinery for continuous deployment can be critical to making your life easier . Automating all this stuff certainly helps me sleep at night much better than I used to when we did manual pushes.