The Case for Gated Releases

At Sefaira, we measure the performance of our product team based on frequency of releases, following the message well captured in this article. I am always making the case for releasing frequently to the point where this is now captured on our meme wall. However, there exists a real conflict between the frequency of releases & the quality of the product that goes out of the door. The tension is fairly predictable - the less frequently we release, the more time we have to improve the quality of the product that goes out (implying better performance, more feature completeness, more polish & less bugs). This tension is real, and troubles many startups including ours

The argument goes like this. If you send out a product with sub-optimal performance, low polish, known areas of improvement, & some undiscovered bugs to the user, you risk getting written off as the user will be disappointed. However, if you spend a significant time improving the product performance, by covering more use cases, adding polish & finding/squashing bugs before the customer gets it, you can ensure that the user will not be disappointed by the product. Who in the right frame of their mind would argue against such sound "user focused" behavior by the product team?

Well, I will and here are reasons why the argument above doesn't apply to most startups (i.e. companies seeking high growth as defined by Paul Graham). There are many unknowns at a startup.

You don't know what your customers want

While you might think you know what your customers want, you have absolutely no knowledge of how much value your users attribute to new features or products. All startups are constantly trying to discover the next product & set of features/functionality/performance improvement which will fuel their growth. This is distinct from optimizing an existing product for your mainstream customers when their desires are absolutely clear. Apple knows its users want higher battery life, and Amazon knows their users want products at a better price, faster (*). On the other hand, Google does not know what its users want from Google Glass, Facebook doesn't know how much its users engage with in-feed video ads (**).

You can't justify product investment based on projections of value delivered

Absent of a quantifiable basis for value of new features, you don't know how much spend can be justified for development of these new features. This leads to significant risk of wasting precious capital on features that don't deliver value to the users & business -  capital that should be used with significant prudence. Note that I say prudence, not sparingly. Sometimes this is misconstrued & product teams end up solving the easy problem that don't require high investment. At Sefaira, when we were were doing early usability tests of real time analysis plugin, we didn't connect the plugin with our cloud based engines. However, we learnt quickly that this could undercut the viability of our product. At this time, we invested time & money in connecting the new plugin with our cloud based engines. Note that the product still lacked polish, which brings me to the last point.

You can't polish your way to success

While polish can amplify engagement of a useful product, polish doesn't convert a useless product into an engaging one. Products without polish are promising at worst, and highly engaging at best. Google Glass is promising, whereas Facebook's early version showed very high engagement. If you compare Mailbox with Boomerang, you see a clear case where polish clearly creates a highly engaging experience with Mailbox whereas poor design with Boomerang keeps its squarely in the "promising" bracket compared to Mailbox.

While the above holds true, this is still not a case for shipping a product that has known areas of improvement, known & unknown bugs & poor polish. Users can get turned off with a poor product. At Sefaira, we surveyed our early users who either did not engage with the product, or didn't come back to use it after 1st experience. Nearly half of these users attributed their lack of engagement to product deficiencies, and almost all of them engaged with the product after our outreach which listed product improvements since their first experience. The key to re-engagement is rapid iteration to add value to users, and ability to reach out to these users with clearly identified improvements. It should be noted that a significant majority of users remained engaged or increased their engagement while we continued to add new users, and more features & performance - an important trait of gated releases.

This brings me to the case for gated releases. I define gated releases, as the roll out of new functionality or product in waves to the entire user base, with well identified metrics that need to be met before the next "gate" is opened & users let in.

Release for learning

Gated releases allow you to learn based on early user experience, and enable meaningful iteration on the product itself. This iteration is based on feedback from a prior batch of users, and unlocks budget for the next gate. Metrics that need to improve are identified & measured. It is important to identify the size of each batch of users such that you can learn meaningfully from it, and ensure that your product team is capable of gathering feedback without getting overwhelmed.

Releases become forgiving

All product teams should take quality assurance seriously. However, there is such a thing as too much testing which can be quantified when you are testing aspects of the product that you are not looking to measure. These aspects are discoverable by you & your users, but simply don't matter for what you are trying to learn. If you roll out such a product through a "gate", you will quickly discover these aspects of your product & you will learn how much it held back user experience. Through the next iteration, you can fix these issues before opening the next "gate". Further, frequent releases allow you to fix mistakes quickly as you can quickly identify source of those mistakes due to the "small delta" nature of gated releases.

Releases unlock budget for next iteration

For all startup product teams, product development should be a series of carefully placed bets. You need to understand what bets to place by sampling your user base as often as possible. Gated releases allow for rapid & meaningful sampling of your user base

Releases eliminate intrigue & win support

A product team that doesn't ship, as well meaning as their reasons might be, can lose stakeholder & customer patience. If you are building it, ship it because shipping delivers value. However, a poor product can kill you. To avoid that fate, roll it out through "gates" of validation

To summarize, our experience has suggested that gated releases provide validated learning and a basis for product investment. It is extremely useful for startups to encourage the practice of gated releases, and it comes with some important pre-requisites

• Infrastructure that enables gated releases
• Development practice that buys into moving fast with focus on value
• Communication to the business that ensures everyone is informed & gathering feedback

* - Amazon still doesn't know how much premium customers attribute to faster shipping, and constantly experiments to determine this. Apple on the other hand can be fairly sure that there is no premium for extra battery life. 

** - Facebook likely knows that its customers attribute significant value to video ads, though only if its users see them.