Rapid releases: one webdev’s perspective

People still seem to be very confused about why Mozilla has moved to the new rapid release system. I thought I’d try and explain it from my perspective. I should point out that I am not any kind of official spokesperson, and should not be quoted as such. The following is just my own personal opinion.

Imagine, now, you work on a team of web developers, and you only get to push new code to production once a year, or once every eighteen months. Your team has decided to wait until the chosen twenty new features are finished, and not ship until those are totally done and passed through a long staging and QA period. The other hundreds of bugs/tickets you closed out in that 12-18 months would have to wait too.

Seems totally foreign in these days of continuous deployment, doesn’t it?

When I first heard about rapid releases, back at our December All Hands, I had two thoughts. The first was that this was absolutely the right thing to do. When stuff is done we should give it to users. We shouldn’t make them wait, especially when other browsers don’t make them wait.

The second thought was that this was completely audacious and I didn’t know if we could pull it off. Amazingly, it happened, and now Mozilla releases use the train model and leave the station every six weeks.

So now users get features shortly after they are done (big win), but there’s been a lot of fallout. Some of the fallout has been around internal tools breaking – we just pushed out a total death sprint release of Socorro to alleviate some of this, for example. Most of the fallout, however, has been external. I see three main areas, and I’ll talk about each one in turn.

Version numbers

The first thing is pushback on version numbers. I see lots of things like:
“Why is Mozilla using marketing-driven version numbers now?”
“What are they trying to prove?”
“How will I know which versions my addons are compatible with?”
“How will I write code (JS/HTML/CSS) that works on a moving target?”

Version numbers are on the way to becoming much less visible in Firefox, like they are in webapps, or in Chrome, for that matter. (As I heard a Microsoft person say, “Nobody asks ‘Which version of Facebook are you running?’”) So to answer: it’s not marketing-driven. In fact, I think not having big versions full of those twenty new features has been much, much harder for the Engagement (marketing) team to know how to market. I see a lot of rage around version numbers in the newsgroups and on tech news sites (HN, Slashdot, etc), which tells me that we haven’t done a good job communicating this to users. I believe this is a communication issue rather than because it’s a bad idea: nowhere do you see these criticisms of Chrome, which uses the same method.

(This blog post is, in part, my way of trying to help with this.)

Add-on compatibility

The add-ons team has been working really hard to minimize add-on breakage. In realistic terms, most add-ons will continue to work with each new release, they just need a version bump. The team has a process for bumping the compatible versions of an add-on automatically, which solves this problem for add-ons that are hosted on addons.mozilla.org. Self-hosted add-ons will continue to need manual updating, and this has caused problems for people.

The goal is, as I understand it, for add-on authors to use the Add-on SDK wherever possible, which will have versions that are stable for a long time. (Read the Add-ons blog on the roadmap for more information on this.)

Enterprise versions

The other thing that’s caused a lot of stress for people at large companies is the idea that versions won’t persist for a long time. Large enterprises tend not to upgrade desktop software frequently. (This is the sad reason why so many people are still on IE6.)

There is an Enterprise Working Group working on these problems: we are taking it seriously.

finally:

Overall, getting Firefox features to users faster is a good thing. Some of the fallout issues were understood well in advance and had a mitigation plan: add-on incompatibility for example. Some others we haven’t done a good job with.

I truly believe if we had continued to release Firefox at the rate of a major version every 18 months or so, that we would have been on a road to nowhere. We had to get faster. It’s a somewhat risky strategy, but it’s better to take that risk than quietly fade away.

At the end of the day we have to remember the Mozilla mission: to promote openness and innovation on the web. It’s hard to promote innovation within the browser if you ship orders of magnitude more slowly than your competitors.

Notice that I mention the mission: people often don’t know or tend to forget that Mozilla isn’t in this for the money. We’re a non-profit. We’re in it for the good of the web, and we want to do whatever’s needed to make the web a better and more open place. We do these things because we’re passionate about the web.

I’ve never worked anywhere else where the mission and the passion were so prominent. We may sometimes do things you don’t agree with, or communicate them in a less-than-optimal way, but I really want people who read this to understand that our intentions are positive and our goal is the same as it’s always been.

18 Comments

  1. Anthony Ricaud:

    About the version numbers and the enterprise, I think we could took inspiration from games. The first release of the year could be 2012.1 and 6 weeks later, 2012.2, etc. Also, 2012.1 would also be called 2012 for the famous LTS version.

    I think it’s very simple to communicate to every group (majority of users, enterprise and web devs) and enables the LTS so many are asking for while not adding that much overhead to Gecko developers (only backporting to a release that is less than 1 year old).

  2. Wladimir Palant:

    Funny thing is, I’ve just written a lengthy explanation on rapid releases in a German-language forum that goes pretty much along the same lines. I see it very similarly to you. Two comments still:

    There are two important reasons (other than communication) why we don’t hear the same criticism for Chrome. Chrome has been doing this from the very start so the people using Chrome are the ones who accepted their rapid releases. And their updates are mostly invisible to the user, they don’t interrupt the workflow (yes, I know that Mozilla is working on that).

    Concerning add-ons, I see this area as the most critical for the rapid releases. While there is some automation to update compatibility ranges on AMO, there are always add-ons that need to be updated manually to work with a release and add-on developers are often doing this too late (for understandable reasons). So people with many add-ons are going to play add-on roulette every six weeks - which is certainly bad for the willingness to accept updates (or install add-ons). Given that add-ons are a major selling point for Firefox this is an issue. And I don’t really see the Add-on SDK solving it.

  3. laura:

    Wladimir: Thanks for the feedback. Obviously you are a lot more knowledgeable about the add-ons situation than I.

    Here’s a question for you: if one assumes rapid releases as a given constraint, what should Mozilla do for add-on authors?

  4. Markus:

    It’s interesting that you mention the Enterprise Working Group. This group was announced with a lot of noise and advance praise a while ago, but since then not a single outcome has been broadly published. On their Wiki-buglist you can find the same old annoying bugs that are now open for several years and where everybody in a company environment is complaining about, and still nobody is working on it. Instead, several man-month seem to go into discussions whether version numbers should be displayed or not. So, what is this enterprise working group doing?

  5. Archaeopteryx:

    The version number rage is overrated if it is about increasing the number, but I would also like to see something different than i++. Removing the version number from about window breaks a common rule in applications. It doesn’t hurt, so “2012.1 (latest version” should be fine.

    For add-ons, fixed freeze dates for APIs and UI and complete documentation of changes are necessary. From my experience, add-on developers are often not aware that beta means API freeze (as far as I know), so they could care about release and beta and only have to update the add-on every 12 weeks. In the past, Mozilla had been marketing/lobbying/pressing the add-on developers early in the release process to update theirs add-ons (iirc already for late alphas), so some developers have bad experiences with caring early about compatibility and seeing the add-on break later for the targeted release.
    I’m guessing Wladimir will write something about binary add-ons.

    Final question: Is there an UI freeze date?

  6. James:

    Amid all the versioning noise of late - I just thought I would point out that it’s probably not entirely accurate to say that the reason many corporate folks are still on IE6 is “just because” it’s not update very frequently - the source of this likely stems to poor choices made back in the days of “Intranets” and ERP software that runs on said Intranets, that uses ugly hacks that are dependent on IE6 being in place. And partly because MS broke their own promises about future proofing their own APIs, so now certain organizations are stuck on IE6 - because they don’t want to spend the $$$ to change the ERP Intranet software to something that will work with a newer more modern browser (good ol’ “ROI” and all that.)

    Frankly, I can see Mozilla facing the same issues down the road. Some companies use unglamorous JavaScript and other language hacks because of “inheiriting” a bug-infested website, for example - in rather the same way, the rapid releases of FF are almost guaranteed to be breaking these websites / web apps.

    Seems to me the outreach to devs on the rapid release cycle for the enterprise (some kind of LTS solution) is going to be one long slog and a lot of “heads ups” to enterprises about what they should be testing to ensure their aren’t breaking changes in “wonky” Intranet / ERP / public and or private websites, especially with those that are public facing.

  7. Wladimir Palant:

    Laura: That’s a good question and I don’t think that I have any real answers. Fact is, the compatiblity center (https://addons.mozilla.org/en-US/firefox/compatibility/) currently shows a phenomenally high level of support for Firefox 6 - 98% is something that we could only dream of for Firefox 4 and earlier releases. But you have to keep in mind that this statistic is weighted by the add-on usage. And some people tend to accumulate obscure add-ons that didn’t even make it into this statistic, those seem to be still likely to break with an update. Given that the same people tend to be very loud in forum discussions, it doesn’t help to instill trust into the update process.

    Maybe Mozilla doesn’t actually need to do something and in a few years the majority of those obscure add-ons will really be using the Add-on SDK (most of them are pretty simple anyway). It would be nice to see some statistics on how many add-ons are already based on the SDK and what the tendency is right now. The main question is whether this change will happen fast enough because the current situation seems pretty devastating for Firefox. Frankly, I don’t really know what Mozilla could do to speed up the process - offer help rewriting the add-ons? In the end, if an add-on isn’t really supported this won’t help. It isn’t a new issue but users are going to hit it more frequently now.

    And then there is the problem of add-ons that are not hosted on AMO. Unfortunately, those are a problem no matter how you take it. I’ve noticed this tendency a while ago already - almost all problem reports that I get and that are caused by other extensions are due to extensions not hosted on AMO. They are typically developed by companies which lack real expertise but don’t even get the rudimentary review that you have on AMO. And: yes, implementing similar mechanisms to what you have on AMO is hard for them. But they wouldn’t want to give away their distribution channels.

  8. Pete:

    The problem is not the rapid release cycle nor the version numbering. In fact it’s bad very communication to a loyal user base: https://bugzilla.mozilla.org/show_bug.cgi?id=678775#c36

    Also no statement of mozilla’s CEO or chairman is heard.

    One can very easily get the impression that mozilla currently has no direction.

  9. Markus:

    I’m surprised that there seems to be kind of a censorship when commenting your blog: Are comments that are not inline with your thinking not published?

  10. laura:

    Wladimir: Thanks for your feedback, you raise some interesting points. (Do you mind if I share them?)

    Pete: Some of the community discussion has been really heated. It’s part of the reason I wanted to post something. Re CEO/Chairman, there was recently posted a vision statement which is worth a read. I’d also encourage you to engage with anyone involved Mozilla all the way up to Mitchell - people are generally pretty approachable.

    Marcus: I just have moderation turned on to control spam (in addition to akismet). I go through and approve stuff daily or so, never had to censor anything that I recall. (I likely would if it was particularly obscene or something, but I’m happy for people to offer differing opinions.)

  11. groovecoder:

    A great post as usual, Laura.

    re: addon compatibility …

    AMO already uses a tool to scan addon code and to give compatible addons the version bump. It seems like the same tool could be used to create bugs for incompatible addons. e.g., “AdBlock Plus uses the preferences.method() call which is deprecated in Firefox 8″

    ?

  12. Wladimir Palant:

    Laura: Feel free to share them with whoever you want. I wish that I had more suggestions. So far this is the only one I could come up with: https://adblockplus.org/blog/on-rapid-releases-and-version-numbers

  13. Bill D:

    Laura - I’m a user, nto an add-on developer - but I’ve pushed Firefox and Thunderbird to family and friends and I keep them all maintained. Where this process has gone wrong is that users simply want their add-ons to work - and my mother doesn’t want a message to say that if she upgrades some of her add-ons will be disabled.
    I’ve got Adobe Acrobat - Create PDF and Java Console disabled in Firefox and Eset NOD32 disabled in Thunderbird. I’ve had to edit install.rdf for Contacts Sidebar to work in Thunderbird - which it does, since it’s just a version number change that was needed.
    I don’t think Mozilla can push this off and say that the add-on developers should update more frequently - you need to find a way to stop breaking the full experience from using my browser and email client every 6-8 weeks.
    Again, as a user, I really like Firefox 4 - but there are no real changes in FF5 or FF6 that make a noticeable difference to me.
    I do hope you find a solution!

  14. Mook:

    To extend your metaphor:

    You are Facebook, except for some reason all of your production machines are not under your control and pushes are staggered beyond your control; also, every time you push, people have to wait for twenty seconds for the page to load. Every time you push to production, all games like Farmville break and can’t be played until their developer gets around to certifying it against the new page. You have plans to auto-certify eventually. There is some unknown but large number of people who have somehow managed to build moderately popular sites of similar games, who you can’t reach, and can’t certify yourself.

    That corresponds, in order, with 1) Mozilla doesn’t control all the installs of Firefox and multiple versions in the wild simultaneously (and time needed to download); 2) addons on AMO; 3) addons off AMO.

  15. Pete:

    Laura, I’m not talking about a vision statement. I’m talking about a CEO who should give a binding direction right now as a reaction of the heated discussion and to show some employees a direction.

    And regarding the hidden version numbers: html5 is a moving target with a) specs coming and going and b) some time until they are implemented into the browser.

    To give you an example: the progress meter elements are now implanted in the latest firefox build. As a web developer, how shall I test if the clients browser supports it or not? For all other browsers except firefox, I can lookup a feature table like caniuse.com and I’m done.

    Any idea how this should work with firefox in the future?

  16. Christopher Blizzard:

    From our Chairwoman: http://blog.lizardwrangler.com/2011/08/25/rapid-release-process/

  17. Angus:

    as an end user the rapid release policy is horsedroppings.

  18. John Thomas:

    To play Devil’s Advocate for a moment, I think one thing that worries people is that adding new features as soon as possible is going to add new bugs. This ties into the idea of major and minor version numbers. It is one thing to change version numbers every week for minor version numbers, bug fixes, minor features, but the idea of major version numbers is that here we are adding features that have not been field tested.

    It is kind of nice to have the option to only start using a feature after it has been field tested. Some of that is now taken up by the Beta/Nightly builds, but there the equation is slightly different since there is no need at stability between something going from Nightly -> Release, while first release version to minor version 1 requires some stability.

Leave a comment