How we reduced our time-to-production by 90%, and quadrupled our velocity

Towards the middle of Q2 2013, I took over the responsibility of leading our product(s) along with Dave who took on the responsibility of managing our development team. For a development team of less than 10 developers, the challenge ahead of us was very clear

• We were doing ~ 5-10 releases a month across our product stack
• Our average elapsed time from conceptualizing a feature or improvement to delivery in production was ~ 100 days

Our mandate was to deliver value to the users faster by accelerating the rate of releases, and drive down time to production as much as we could. Essentially, we wanted to become more responsive to the needs of our customer & the business

The Old World

While there were many areas of improvement, I list what we noticed as top characteristics of our development process

• Developing user stories, use cases & scenarios - Our product management served as strong proxy for the customer & diligently detailed user stories & use cases to help the developers learn of our customers’ needs. This was the reasonable thing to do as our product requires significant input from domain experts, and prepping use cases to help developers understand the problems our customers want to solve.
• Strong competence in specific areas of our stack - Our developers had cultivated significant competence across clearly identified areas of our development stack. Again, like many teams, we matched skills to areas of code. This allowed us to move the respective areas of code faster & faster over time
• Quality assurance by QA team & product management before shipping - We had developed a culture of quality assurance as the last gate of development before shipping to the customer. This allowed us to catch areas where we hadn’t met our customers needs, or the user story wasn’t completely satisfied before shipping the product. This is difficult to argue against.

As strange as it may sound, these were the very practices that held us back. Let’s look at them up close

• Developing user stories, use cases & scenarios – On face, this is a very good idea. However, this abstracted our developers from the customers which made it difficult for our developers to understand the pains of our customers. The answer here was clear. We needed to move our product team’s center of gravity closer to the customer, and eliminate product management as a bottleneck. Instead, product management would serve as a bridge to the customer, not a gatekeeper.
• Strong competence in specific areas of our stack – This worked very well for us in the early days. However, given that everything we ship is cross functional, we needed developers from every area of code to ship anything. This led to key man dependencies that bottlenecked our development efforts.
• Quality assurance by QA team & product management before shipping – Since product management had the best understanding of our users’ problems, this was considered an important & critical step. Product management would request domain experts on our team to validate the product against expected behavior prior to shipping. This led to significant time added after development & prior to shipping.

The New World

We developed principles that we would follow to break away from this practice & reduce our time to production while increasing our velocity (as measured by # of releases)

• Move our center of gravity closer to our customer – The purpose here was singular. We needed our development team to have a visceral understanding of our customer’s goals & pains. We did this by holding multiple seminars on architect’s workflow as well as building physics in-house. In parallel, we started conducting user interviews & usability sessions – the caveat being that the latter would be led by our developers, coordinated by product management
• Reduce batch size – We decided that we would work relentlessly to minimize the quantum of product enhancement that can be meaningfully sent to our customers. We also made commensurate moves across our technical stack to facilitate this
• Decentralize releases – We wanted developers to be responsible for shipping the product, right from the stage of conceptualizing the ticket to release. This could only be done if we reduced the size of each release, product wise, to something that could be managed by a developer. Further, it required our services to move independently of each other when needed.
• Enhance code coverage – We decided to make conscious effort, even at the cost of near term benefit, of sharing code across the team. This was done to allow multiple developers to move code across the stack.

The result, thus far

Time to production vs. creation date

• We drove down our time to production (defined as time from conceptualization to production) from 100 days on average to 10 days. As a result, we are now very responsive both to our customers' needs and our vision for the product.

# of releases per month

• We have nearly quadrupled the number of releases we do. This really paid off for us in Q4 where we launched 2 new products – Sefaira for SketchUp & Sefaira for Revit, and did multiple releases which benefited our customers.
• In terms of code coverage, 80% of our developers contributed to areas of our code where we invested 80% of development budget

While many challenges lie ahead of us in terms of driving growth, it is encouraging to note that we are accelerating our product development. Our customers & investors expect nothing less.

The Case for Gated Releases

At Sefaira, we measure the performance of our product team based on frequency of releases, following the message well captured in this article. I am always making the case for releasing frequently to the point where this is now captured on our meme wall. However, there exists a real conflict between the frequency of releases & the quality of the product that goes out of the door. The tension is fairly predictable - the less frequently we release, the more time we have to improve the quality of the product that goes out (implying better performance, more feature completeness, more polish & less bugs). This tension is real, and troubles many startups including ours

The argument goes like this. If you send out a product with sub-optimal performance, low polish, known areas of improvement, & some undiscovered bugs to the user, you risk getting written off as the user will be disappointed. However, if you spend a significant time improving the product performance, by covering more use cases, adding polish & finding/squashing bugs before the customer gets it, you can ensure that the user will not be disappointed by the product. Who in the right frame of their mind would argue against such sound "user focused" behavior by the product team?

Well, I will and here are reasons why the argument above doesn't apply to most startups (i.e. companies seeking high growth as defined by Paul Graham). There are many unknowns at a startup.

You don't know what your customers want

While you might think you know what your customers want, you have absolutely no knowledge of how much value your users attribute to new features or products. All startups are constantly trying to discover the next product & set of features/functionality/performance improvement which will fuel their growth. This is distinct from optimizing an existing product for your mainstream customers when their desires are absolutely clear. Apple knows its users want higher battery life, and Amazon knows their users want products at a better price, faster (*). On the other hand, Google does not know what its users want from Google Glass, Facebook doesn't know how much its users engage with in-feed video ads (**).

You can't justify product investment based on projections of value delivered

Absent of a quantifiable basis for value of new features, you don't know how much spend can be justified for development of these new features. This leads to significant risk of wasting precious capital on features that don't deliver value to the users & business -  capital that should be used with significant prudence. Note that I say prudence, not sparingly. Sometimes this is misconstrued & product teams end up solving the easy problem that don't require high investment. At Sefaira, when we were were doing early usability tests of real time analysis plugin, we didn't connect the plugin with our cloud based engines. However, we learnt quickly that this could undercut the viability of our product. At this time, we invested time & money in connecting the new plugin with our cloud based engines. Note that the product still lacked polish, which brings me to the last point.

You can't polish your way to success

While polish can amplify engagement of a useful product, polish doesn't convert a useless product into an engaging one. Products without polish are promising at worst, and highly engaging at best. Google Glass is promising, whereas Facebook's early version showed very high engagement. If you compare Mailbox with Boomerang, you see a clear case where polish clearly creates a highly engaging experience with Mailbox whereas poor design with Boomerang keeps its squarely in the "promising" bracket compared to Mailbox.

While the above holds true, this is still not a case for shipping a product that has known areas of improvement, known & unknown bugs & poor polish. Users can get turned off with a poor product. At Sefaira, we surveyed our early users who either did not engage with the product, or didn't come back to use it after 1st experience. Nearly half of these users attributed their lack of engagement to product deficiencies, and almost all of them engaged with the product after our outreach which listed product improvements since their first experience. The key to re-engagement is rapid iteration to add value to users, and ability to reach out to these users with clearly identified improvements. It should be noted that a significant majority of users remained engaged or increased their engagement while we continued to add new users, and more features & performance - an important trait of gated releases.

This brings me to the case for gated releases. I define gated releases, as the roll out of new functionality or product in waves to the entire user base, with well identified metrics that need to be met before the next "gate" is opened & users let in.

Release for learning

Gated releases allow you to learn based on early user experience, and enable meaningful iteration on the product itself. This iteration is based on feedback from a prior batch of users, and unlocks budget for the next gate. Metrics that need to improve are identified & measured. It is important to identify the size of each batch of users such that you can learn meaningfully from it, and ensure that your product team is capable of gathering feedback without getting overwhelmed.

Releases become forgiving

All product teams should take quality assurance seriously. However, there is such a thing as too much testing which can be quantified when you are testing aspects of the product that you are not looking to measure. These aspects are discoverable by you & your users, but simply don't matter for what you are trying to learn. If you roll out such a product through a "gate", you will quickly discover these aspects of your product & you will learn how much it held back user experience. Through the next iteration, you can fix these issues before opening the next "gate". Further, frequent releases allow you to fix mistakes quickly as you can quickly identify source of those mistakes due to the "small delta" nature of gated releases.

Releases unlock budget for next iteration

For all startup product teams, product development should be a series of carefully placed bets. You need to understand what bets to place by sampling your user base as often as possible. Gated releases allow for rapid & meaningful sampling of your user base

Releases eliminate intrigue & win support

A product team that doesn't ship, as well meaning as their reasons might be, can lose stakeholder & customer patience. If you are building it, ship it because shipping delivers value. However, a poor product can kill you. To avoid that fate, roll it out through "gates" of validation

To summarize, our experience has suggested that gated releases provide validated learning and a basis for product investment. It is extremely useful for startups to encourage the practice of gated releases, and it comes with some important pre-requisites

• Infrastructure that enables gated releases
• Development practice that buys into moving fast with focus on value
• Communication to the business that ensures everyone is informed & gathering feedback

* - Amazon still doesn't know how much premium customers attribute to faster shipping, and constantly experiments to determine this. Apple on the other hand can be fairly sure that there is no premium for extra battery life. 

** - Facebook likely knows that its customers attribute significant value to video ads, though only if its users see them.

One use case that validated my decision to switch to Android

Like many, I became an iPhone user many moons ago because I didn't want to miss out on a device revolution. After being an iPhone user for 5+ years, I had a sneaking suspicion that I was missing out on a similar revolution - one that was blurring the lines between the device & the internet. So I switched to an Android phone (the device doesn't matter, so it shall stay anonymous)

Sure, Android does some things that I can best describe as ranging from creepy (automatically processing images through Goggles) to nifty (Google Now gives me timely information on weather, local events, etc). However, one use case validated my decision to switch to Android.

I use Open Table for dinner reservations. Back in the iPhone days, I would make dinner reservations, and then set up a calendar invite with the reservation. I relied on my habit of looking at the calendar every few hours to remember that I had a dinner reservation. I would check how far the restaurant was, plan the route & figure out how far ahead i needed to leave to get to dinner on time. Of course, as I was pulling into the restaurant, I would get a completely worthless reminder from my calendar.

This is what the same use case looks like with Android. I make a dinner reservation on Open Table. Since Google scans my email, it knows that I have a reservation at a restaurant on a given date & time, so instead of adding it to the calendar, it simply shows up on Google Now on the day of reservation. Further, it knows where the reservation is, so it reminds me to leave for dinner based on the commute time to the restaurant including a route plan. Finally, I don't get an annoying reminder 10 minutes before I pull into the restaurant.

This blurring of lines between the device & the internet is why I think Android will eat iOS. I tell people that Android taps into a greater ecosystem, and I get quizzical looks as people point me to the iOS app store which I construe as a very narrow definition of ecosystem. Yes, iOS has an amazing app store and Android is staying close. However, when it comes to tapping into the internet is concerned, iOS isn't even close.

Building better products through balanced inquiry

Like many other product companies, we have a relentless focus on moving our product team's center of gravity outside the building. We do so by doing frequent usability testing, and are currently maintaining 1 session/week. 

A challenge we often grapple with, when it comes to usability tests, is the conflict between structuring our questions which allows contextual inquiry, vs. a free flowing nature of conversation which allows us to discover areas we were blind to. Our view is that both of these modes are important, and must co-exist. Here is why.

Benefits of structured inquiry

Structured inquiry allows product teams to validate the product vision. These are often answers to problems that the user hasn't discovered yet or didn't consider possible. This is representative of the famous quote (allegedly) from Henry Ford -  "If I had asked people what they wanted, they would have said faster horses", to suggest that people couldn't imagine that they needed a car. Improvement of the product along this line leads to disruption in user work flow, & adoption often by the "irrational" user (due to novelty of the product). For us at Sefaira, this included:

• Real-time analysis - Our users knew that building performance analysis takes several minutes to hours, that they would've never expected to see results in seconds. This changes their work flow & behavior in ways they didn't expect
• Feedback as you design - Our users were used to performing energy analysis in separate specialized analysis software, and didn't consider integration with their design environments as a possibility
• Results first paradigm - All current analysis software forces users to follow a workflow of specifying all inputs & then examining results - a classic design to performance paradigm. Users couldn't imagine that a paradigm which flipped this (performance to design) could even exist.

It would be fair to state that this helps us discover "global optimum"

Benefits of free flowing inquiry

Free flowing inquiry allows product teams to uncover their blind spots. These are often discoveries about the user's patterns & work flows that the product team was blind to, or had assumed during product development. Improvement of the product along this line leads to greater adoption by the "mainstream" or "rational" user. For us at Sefaira, this led us to discover the importance of many things we were ignorant of

• Widespread use of collections & layers in SketchUp
• Widespread (mis)use of custom families in Revit
• Arduous nature of information management in building design software

This helps us discover "local optimum".

Conclusion

Balanced inquiry during usability testing leads to realization of product vision while ensuring deeper adoption of the product by the mainstream user. Product teams can balance inquiry by doing the following

• Note down 3-5 things we are trying to learn about the product
• Give users a lot of white space to talk freely, and echo their questions back to them
• Ask them what they would build next in the product, & why

I encourage product teams to identify benefits their users have realized & features built in their product, and correlate (as best as possible) those improvements to engagement & revenue metrics of the product. This presents a meaningful view of the trajectory of your product & its impact on business.