How we reduced our time-to-production by 90%, and quadrupled our velocity

Towards the middle of Q2 2013, I took over the responsibility of leading our product(s) along with Dave who took on the responsibility of managing our development team. For a development team of less than 10 developers, the challenge ahead of us was very clear

• We were doing ~ 5-10 releases a month across our product stack
• Our average elapsed time from conceptualizing a feature or improvement to delivery in production was ~ 100 days

Our mandate was to deliver value to the users faster by accelerating the rate of releases, and drive down time to production as much as we could. Essentially, we wanted to become more responsive to the needs of our customer & the business

The Old World

While there were many areas of improvement, I list what we noticed as top characteristics of our development process

• Developing user stories, use cases & scenarios - Our product management served as strong proxy for the customer & diligently detailed user stories & use cases to help the developers learn of our customers’ needs. This was the reasonable thing to do as our product requires significant input from domain experts, and prepping use cases to help developers understand the problems our customers want to solve.
• Strong competence in specific areas of our stack - Our developers had cultivated significant competence across clearly identified areas of our development stack. Again, like many teams, we matched skills to areas of code. This allowed us to move the respective areas of code faster & faster over time
• Quality assurance by QA team & product management before shipping - We had developed a culture of quality assurance as the last gate of development before shipping to the customer. This allowed us to catch areas where we hadn’t met our customers needs, or the user story wasn’t completely satisfied before shipping the product. This is difficult to argue against.

As strange as it may sound, these were the very practices that held us back. Let’s look at them up close

• Developing user stories, use cases & scenarios – On face, this is a very good idea. However, this abstracted our developers from the customers which made it difficult for our developers to understand the pains of our customers. The answer here was clear. We needed to move our product team’s center of gravity closer to the customer, and eliminate product management as a bottleneck. Instead, product management would serve as a bridge to the customer, not a gatekeeper.
• Strong competence in specific areas of our stack – This worked very well for us in the early days. However, given that everything we ship is cross functional, we needed developers from every area of code to ship anything. This led to key man dependencies that bottlenecked our development efforts.
• Quality assurance by QA team & product management before shipping – Since product management had the best understanding of our users’ problems, this was considered an important & critical step. Product management would request domain experts on our team to validate the product against expected behavior prior to shipping. This led to significant time added after development & prior to shipping.

The New World

We developed principles that we would follow to break away from this practice & reduce our time to production while increasing our velocity (as measured by # of releases)

• Move our center of gravity closer to our customer – The purpose here was singular. We needed our development team to have a visceral understanding of our customer’s goals & pains. We did this by holding multiple seminars on architect’s workflow as well as building physics in-house. In parallel, we started conducting user interviews & usability sessions – the caveat being that the latter would be led by our developers, coordinated by product management
• Reduce batch size – We decided that we would work relentlessly to minimize the quantum of product enhancement that can be meaningfully sent to our customers. We also made commensurate moves across our technical stack to facilitate this
• Decentralize releases – We wanted developers to be responsible for shipping the product, right from the stage of conceptualizing the ticket to release. This could only be done if we reduced the size of each release, product wise, to something that could be managed by a developer. Further, it required our services to move independently of each other when needed.
• Enhance code coverage – We decided to make conscious effort, even at the cost of near term benefit, of sharing code across the team. This was done to allow multiple developers to move code across the stack.

The result, thus far

Time to production vs. creation date

• We drove down our time to production (defined as time from conceptualization to production) from 100 days on average to 10 days. As a result, we are now very responsive both to our customers' needs and our vision for the product.

# of releases per month

• We have nearly quadrupled the number of releases we do. This really paid off for us in Q4 where we launched 2 new products – Sefaira for SketchUp & Sefaira for Revit, and did multiple releases which benefited our customers.
• In terms of code coverage, 80% of our developers contributed to areas of our code where we invested 80% of development budget

While many challenges lie ahead of us in terms of driving growth, it is encouraging to note that we are accelerating our product development. Our customers & investors expect nothing less.

The Case for Gated Releases

At Sefaira, we measure the performance of our product team based on frequency of releases, following the message well captured in this article. I am always making the case for releasing frequently to the point where this is now captured on our meme wall. However, there exists a real conflict between the frequency of releases & the quality of the product that goes out of the door. The tension is fairly predictable - the less frequently we release, the more time we have to improve the quality of the product that goes out (implying better performance, more feature completeness, more polish & less bugs). This tension is real, and troubles many startups including ours

The argument goes like this. If you send out a product with sub-optimal performance, low polish, known areas of improvement, & some undiscovered bugs to the user, you risk getting written off as the user will be disappointed. However, if you spend a significant time improving the product performance, by covering more use cases, adding polish & finding/squashing bugs before the customer gets it, you can ensure that the user will not be disappointed by the product. Who in the right frame of their mind would argue against such sound "user focused" behavior by the product team?

Well, I will and here are reasons why the argument above doesn't apply to most startups (i.e. companies seeking high growth as defined by Paul Graham). There are many unknowns at a startup.

You don't know what your customers want

While you might think you know what your customers want, you have absolutely no knowledge of how much value your users attribute to new features or products. All startups are constantly trying to discover the next product & set of features/functionality/performance improvement which will fuel their growth. This is distinct from optimizing an existing product for your mainstream customers when their desires are absolutely clear. Apple knows its users want higher battery life, and Amazon knows their users want products at a better price, faster (*). On the other hand, Google does not know what its users want from Google Glass, Facebook doesn't know how much its users engage with in-feed video ads (**).

You can't justify product investment based on projections of value delivered

Absent of a quantifiable basis for value of new features, you don't know how much spend can be justified for development of these new features. This leads to significant risk of wasting precious capital on features that don't deliver value to the users & business -  capital that should be used with significant prudence. Note that I say prudence, not sparingly. Sometimes this is misconstrued & product teams end up solving the easy problem that don't require high investment. At Sefaira, when we were were doing early usability tests of real time analysis plugin, we didn't connect the plugin with our cloud based engines. However, we learnt quickly that this could undercut the viability of our product. At this time, we invested time & money in connecting the new plugin with our cloud based engines. Note that the product still lacked polish, which brings me to the last point.

You can't polish your way to success

While polish can amplify engagement of a useful product, polish doesn't convert a useless product into an engaging one. Products without polish are promising at worst, and highly engaging at best. Google Glass is promising, whereas Facebook's early version showed very high engagement. If you compare Mailbox with Boomerang, you see a clear case where polish clearly creates a highly engaging experience with Mailbox whereas poor design with Boomerang keeps its squarely in the "promising" bracket compared to Mailbox.

While the above holds true, this is still not a case for shipping a product that has known areas of improvement, known & unknown bugs & poor polish. Users can get turned off with a poor product. At Sefaira, we surveyed our early users who either did not engage with the product, or didn't come back to use it after 1st experience. Nearly half of these users attributed their lack of engagement to product deficiencies, and almost all of them engaged with the product after our outreach which listed product improvements since their first experience. The key to re-engagement is rapid iteration to add value to users, and ability to reach out to these users with clearly identified improvements. It should be noted that a significant majority of users remained engaged or increased their engagement while we continued to add new users, and more features & performance - an important trait of gated releases.

This brings me to the case for gated releases. I define gated releases, as the roll out of new functionality or product in waves to the entire user base, with well identified metrics that need to be met before the next "gate" is opened & users let in.

Release for learning

Gated releases allow you to learn based on early user experience, and enable meaningful iteration on the product itself. This iteration is based on feedback from a prior batch of users, and unlocks budget for the next gate. Metrics that need to improve are identified & measured. It is important to identify the size of each batch of users such that you can learn meaningfully from it, and ensure that your product team is capable of gathering feedback without getting overwhelmed.

Releases become forgiving

All product teams should take quality assurance seriously. However, there is such a thing as too much testing which can be quantified when you are testing aspects of the product that you are not looking to measure. These aspects are discoverable by you & your users, but simply don't matter for what you are trying to learn. If you roll out such a product through a "gate", you will quickly discover these aspects of your product & you will learn how much it held back user experience. Through the next iteration, you can fix these issues before opening the next "gate". Further, frequent releases allow you to fix mistakes quickly as you can quickly identify source of those mistakes due to the "small delta" nature of gated releases.

Releases unlock budget for next iteration

For all startup product teams, product development should be a series of carefully placed bets. You need to understand what bets to place by sampling your user base as often as possible. Gated releases allow for rapid & meaningful sampling of your user base

Releases eliminate intrigue & win support

A product team that doesn't ship, as well meaning as their reasons might be, can lose stakeholder & customer patience. If you are building it, ship it because shipping delivers value. However, a poor product can kill you. To avoid that fate, roll it out through "gates" of validation

To summarize, our experience has suggested that gated releases provide validated learning and a basis for product investment. It is extremely useful for startups to encourage the practice of gated releases, and it comes with some important pre-requisites

• Infrastructure that enables gated releases
• Development practice that buys into moving fast with focus on value
• Communication to the business that ensures everyone is informed & gathering feedback

* - Amazon still doesn't know how much premium customers attribute to faster shipping, and constantly experiments to determine this. Apple on the other hand can be fairly sure that there is no premium for extra battery life. 

** - Facebook likely knows that its customers attribute significant value to video ads, though only if its users see them.

One use case that validated my decision to switch to Android

Like many, I became an iPhone user many moons ago because I didn't want to miss out on a device revolution. After being an iPhone user for 5+ years, I had a sneaking suspicion that I was missing out on a similar revolution - one that was blurring the lines between the device & the internet. So I switched to an Android phone (the device doesn't matter, so it shall stay anonymous)

Sure, Android does some things that I can best describe as ranging from creepy (automatically processing images through Goggles) to nifty (Google Now gives me timely information on weather, local events, etc). However, one use case validated my decision to switch to Android.

I use Open Table for dinner reservations. Back in the iPhone days, I would make dinner reservations, and then set up a calendar invite with the reservation. I relied on my habit of looking at the calendar every few hours to remember that I had a dinner reservation. I would check how far the restaurant was, plan the route & figure out how far ahead i needed to leave to get to dinner on time. Of course, as I was pulling into the restaurant, I would get a completely worthless reminder from my calendar.

This is what the same use case looks like with Android. I make a dinner reservation on Open Table. Since Google scans my email, it knows that I have a reservation at a restaurant on a given date & time, so instead of adding it to the calendar, it simply shows up on Google Now on the day of reservation. Further, it knows where the reservation is, so it reminds me to leave for dinner based on the commute time to the restaurant including a route plan. Finally, I don't get an annoying reminder 10 minutes before I pull into the restaurant.

This blurring of lines between the device & the internet is why I think Android will eat iOS. I tell people that Android taps into a greater ecosystem, and I get quizzical looks as people point me to the iOS app store which I construe as a very narrow definition of ecosystem. Yes, iOS has an amazing app store and Android is staying close. However, when it comes to tapping into the internet is concerned, iOS isn't even close.

Building better products through balanced inquiry

Like many other product companies, we have a relentless focus on moving our product team's center of gravity outside the building. We do so by doing frequent usability testing, and are currently maintaining 1 session/week. 

A challenge we often grapple with, when it comes to usability tests, is the conflict between structuring our questions which allows contextual inquiry, vs. a free flowing nature of conversation which allows us to discover areas we were blind to. Our view is that both of these modes are important, and must co-exist. Here is why.

Benefits of structured inquiry

Structured inquiry allows product teams to validate the product vision. These are often answers to problems that the user hasn't discovered yet or didn't consider possible. This is representative of the famous quote (allegedly) from Henry Ford -  "If I had asked people what they wanted, they would have said faster horses", to suggest that people couldn't imagine that they needed a car. Improvement of the product along this line leads to disruption in user work flow, & adoption often by the "irrational" user (due to novelty of the product). For us at Sefaira, this included:

• Real-time analysis - Our users knew that building performance analysis takes several minutes to hours, that they would've never expected to see results in seconds. This changes their work flow & behavior in ways they didn't expect
• Feedback as you design - Our users were used to performing energy analysis in separate specialized analysis software, and didn't consider integration with their design environments as a possibility
• Results first paradigm - All current analysis software forces users to follow a workflow of specifying all inputs & then examining results - a classic design to performance paradigm. Users couldn't imagine that a paradigm which flipped this (performance to design) could even exist.

It would be fair to state that this helps us discover "global optimum"

Benefits of free flowing inquiry

Free flowing inquiry allows product teams to uncover their blind spots. These are often discoveries about the user's patterns & work flows that the product team was blind to, or had assumed during product development. Improvement of the product along this line leads to greater adoption by the "mainstream" or "rational" user. For us at Sefaira, this led us to discover the importance of many things we were ignorant of

• Widespread use of collections & layers in SketchUp
• Widespread (mis)use of custom families in Revit
• Arduous nature of information management in building design software

This helps us discover "local optimum".

Conclusion

Balanced inquiry during usability testing leads to realization of product vision while ensuring deeper adoption of the product by the mainstream user. Product teams can balance inquiry by doing the following

• Note down 3-5 things we are trying to learn about the product
• Give users a lot of white space to talk freely, and echo their questions back to them
• Ask them what they would build next in the product, & why

I encourage product teams to identify benefits their users have realized & features built in their product, and correlate (as best as possible) those improvements to engagement & revenue metrics of the product. This presents a meaningful view of the trajectory of your product & its impact on business.

Importance of listing reasons for NOT building the new feature

There are many aspects to building a successful product including the product roadmap & pricing of new features commensurate to investment. While building the roadmap, it is important to consider sequencing as sequential synergies can help you unlock subsequent opportunities as lower cost. This is to say that timing matters as much as features on roadmap. When a certain feature is timed right, it will deliver significant value, & when ill-timed, it will become a liability & possibly sink a young product.

My sense is that most product managers get the ideas part of the roadmap right, thanks to diverse stream of information from sales to customer success to customers themselves. However, they struggle with "right-timing" these features. This problems scales all the way from establishing a roadmap for the next year to establishing priorities for the next sprint.

When trying to determine when to build (be it next sprint, next month or next quarter), I find that it is equally valuable to list why you shouldn't build a certain feature now. This enables healthy reflection from the negative space, and makes the prioritization & timing problem more information complete.

In addition, here are some other signals that you are prioritizing for the wrong reasons

• If you find yourself saying "We need to build it eventually, so why not now?", this is likely a bad reason
• If you find yourself saying "How much effort can it be to build this?", you are likely building it for the wrong reason & of course ignoring total cost of ownership of whatever you are building
• If you find yourself saying "Stakeholder X will be happy if we build this", you are likely prioritizing for the wrong reason. At this time, it is best to have a conversation with Mr. X
• If you find yourself saying "It will make some future important item easier to build", you are likely prioritizing for the wrong reason. Note, if you find yourself saying "it will ALSO make the future important item easier to build", you are finding additional basis for an already sound priority.

Kill the Product Backlog

Sometimes nothing can be a real cool hand - Paul Newman in Cool Hand Luke

Like many startups, at Sefaira we are organized with the goal of reducing the lead-time from feature conception to feature delivery. This forces us to constantly evaluate our product development process & tools, with a clear goal in mind. Amongst many virtues that underpin this goal (measured as time to production), is the idea that shorter lead-time allows the product team to be more reactive and nimble (yes, Agile) to customer needs.

Typically (i.e. not in product discovery mode), product managers understand customer needs, create user stories & use cases, discuss them with the development team to help the team understand the problem, followed by estimation and scoping discussions. Following this, the features are broken in multiple customer facing releases. Good product managers identify data they’d look for before building features, i.e. good old hypotheses driven iterations. New features are often balanced with improvements, bug fixes as well as technical improvements, and prioritized by the product manager.

To manage all this seeming chaos, &  queue up work, some product teams lean on creating and maintaining backlogs. This is a great idea – the product manager logs & maintains all disparate items, sorts and sifts through the backlog working closely with other stakeholders, and lines up work for the development team, and good development teams crank out code and deliver features. If the business wants to know what’s coming up, the product manager leans on the backlog, which is supposed to be a reflection of the product outlook 2-3 months out. This is comforting, sturdy and has the signs of a well-oiled machine. So where’s the problem?

There are 3 key problems here.

Backlogs substitute judgment

Good product managers are always on the lookout for new information to influence product direction. This is intended to influence features to build, bugs to fix, areas to invest in, etc. As new information appears, the product manager with a well-organized backlog goes ahead and prioritizes work in the backlog based on this information. This is where the problem starts. 

The backlog was created when the said new information wasn’t available. While this seems reasonable, what is really happening is that the product manager is using new information to confirm existing work queue. This makes an agile team less reactive to customer needs! What should have happened is that the new information, in concert with P+L goals of product & product vision be used to create new work – as often as necessary. This is discomforting, difficult and keeps me up at night. And if I am uncomfortable, I know that I am exercising judgment. I work with stakeholders to queue up work every sprint based on product outlook (more on this below) and latest information.

Backlogs are cognitive black holes

Once a backlog exists, maintaining it, defending it and justifying it takes up significant cognitive effort. It also becomes a basis for defending product decisions. One of the frequent responses to new requests is “It’s in the backlog”, and this is interpreted as “We got it”. Backlog gradually ends up becoming one of the measures of competence and organization of a product team. It is not hard to see why this is an easy trap to fall into. Backlogs also sap significant cognitive resources – resources that could be brought to bear in gathering & organizing new information with intent to act in immediate future.

Backlogs are poor predictors of product outlook

When asked what we are going to build for our customers in the next 2-3 months, backlogs seem to be the natural place to go look for answers. After all, backlogs are a manifestation of product roadmap and priorities. But if you think deeper about it, product outlook should be based on most recent information that the product team has. Product backlogs by definition are historic. Therein lies the dissonance between product outlook & product backlog. I think product managers should update and maintain product outlook very actively (once every 2 weeks), and update all stakeholders regularly.

Now that I have made the case for killing the backlog, there is a natural question that arises. How do I make sure nothing falls through the cracks? The answer here is simple – stuff will fall through the cracks, and that’s okay! Any fast moving development team (where code is in a good state & movable) can move quickly to pick up items that fall through cracks and deliver it quickly. Its better to have things falling through cracks and have someone notice, than overbuilding or building based on historic information ala the backlog.

Till then, I leave you with this great scene from one of my favorite films “Cool Hand Luke”.

It takes three to tango

In the age of SaaS, lean and a world where rate of improvement of technology is outpacing the rate at which development processes can catch up, understanding roles of various stakeholders in a software development organization becomes more critical than it has ever been.

For building a successful product, there are three key stakeholders who need to work closely together. These are the CEO, the head of product, and the head of software development. All three have unique perspectives, differing priorities, and different roles to play in order to make the product & business successful. This post was motivated by our mission at Sefaira, as well as posts by visionaries such as Ben Horowitz.

Head of development

• Asset quality - Code is a software company's top asset (some would say customers are, and I would agree but this is a chicken & egg scenario), and the product is an extension of the code. There are many other things that play a role in how a user experiences the product, but it turns strongly on the quality of code. I see high quality as a very important goal for head of development. This implies everything from right approach to testing, understanding requirements, ensuring collaboration between developers, etc. Further, I view asset quality & velocity as virtuous traits, and that there is no tension between these two.
• Deployability of assets - In line with the lean approach, especially relevant for start ups, is the ability to deploy code quickly. This implies that code should be movable & deployable at very short notice based on business needs. This requires very good architecture, which starts with accepting that business needs can change, and assets shouldn't become brittle. This ranges from architecture itself, and includes technology choices, language choices as well as infrastructure choices.
• Lead time to creation of assets - How quickly a team goes from raw materials to shipped product is a key KPI for a development team. Assets by definition need to be shipped, otherwise its just inventory which is a liability for software teams. (We have all heard of feature-branch management, and how this can sap valuable time).

Head of product

• P+L of product - Head of product needs to absolutely make sure that all decisions are weighed by cost-benefit analysis (first & life cycle), and benefit is defined by core metrics, which can vary from company to company (market share vs. revenue or some mix of the two). For SaaS companies, engagement is vital & can be defined by new customers using & paying for the product, as well as existing customers increasing time spent using the product continues. This is a key leading indicator.
• Realizing the product vision - Coming up with strategies such that the product roadmap is always in sync with the product vision, and enacting product vision while maintaining a watchful eye on P+L of the product is key. This becomes very critical if there is tension between what sales teams might need, vs. the product vision. Breaking this down further, the head of product needs to keep an eye on day to day execution in terms of requirements of product being translated into assets, constantly ensuring that the product doesn't creep away from original intent.
• Business needs vs. state of assets - It is vital to ensure that assets are being created, & moved to fulfill business needs, and this is the responsibility of head of product. Further providing visibility to the business on the state of assets, and the direction being taken with assets (in terms of how this is the right thing for business needs). In order to do this, head of product is constantly investing in small bets, always course correcting to minimize the gap between what the business needs, and what assets exist & their deployability.

The CEO

• Standard bearer for product vision - The CEO is the standard bearer for product vision for most start ups. The CEO maintains and encourages the drive towards realizing product vision. She enables the head of product to drive towards execution of vision, always ensuring that there is a viable strategy of reconciling short term business needs with long term vision.
• Voice of the business - Before you cry out loud about how the head of product is the voice of business, read on. With goals of heads of product & development stated above, the CEO needs to spell out what the business needs from product to enable day to day execution. The CEO understands (and should understand) the customer journey end to end, and will know what various functions need from the product to execute successfully. A simple example of this is integration of customer engagement platforms such as Totango with the product, to drive customer segmentation & success.
• Magic - I don't know what to call this, or how to quantify this in terms of core metrics but this is pure CEO craftsmanship ranging from aspects of customer experience to good citizenship. I welcome examples of this. Absent of better classification, we'll just call them "nobody's problems but the CEO's"

I wonder how other start ups organize around these stakeholders as they pursue their mission.