Decentralizing product management for shorter cycle times

For any enterprise software company or startup, the ultimate measure of success is how much money it makes for its investors. Money is the best surrogate we have for value a company delivers to the world - and by extension, the impact it has on the world. 

For a product company like ours, its the product that delivers this value. Hence, the goal of product management is to build products & features that customers value & are willing to pay for. Our customers derive value when they use our products to address important design problems, but the journey starts much earlier, starting with how they discover our product, their perception of the product pricing, their experience during the sales process, and most importantly - their experience after they have bought the product.

On the other side, in order to be successful, product management works with designers & software developers in defining and sequencing what needs to be built to deliver value.

Development cycles are no longer a problem area

While the latter has received significant attention, and almost down to a science. Thanks in part to the rich literature on development, over the last year we have continued to drive down our time-to-production, which today stands at 5 days (now with much reduced variability since we moved away from Scrum & adopted Kanban). This means that any new feature that customers would value can be built & deployed in less than 5 days. A major contributor here has also been our focus on moving developers closer to customers via support rotations, frequent usability testing, and integration with engagement-tracking software like Totango which give us rich visibility into how the product is being used. We don't ship any new feature without tracking commensurate module/action in Totango.

Time to revenue as a metric

While for software development teams Time to production metric an objective works well, one of the metrics to track for product management becomes Time to Revenue, and minimizing this time requires significant focus in ensuring that information flows from the edges of the business to product management, and the other way around, as quickly as possible.

Now unlike software developers and designers, their primary job is not to respond to product management's needs. Many constituents of the business interact with the world, and their primary job ranges from marketing & selling the product to helping customers adopt the product in greater numbers, thereby making money for the business. This makes flow of information inherently frictional, and the business must organize to make this as frictionless as possible

In order to solve this problem, most companies have product managers who go & speak with customers, sales teams, marketing teams, and customer success teams to learn about the world outside, and in turn helps them do their job better by arming them with valuable information. This makes product managers an important constituent. This also implies that there is reduced burden on other business teams in deeply understanding the needs of product management.

Decentralizing product management

At Sefaira, we have taken a different approach. We believe that organizing our teams to maximize our surface area that interacts with the world, product management needs to be distributed at the edges, and product management responsibilities need to be embedded with the edges of the business. In order to be successful here, we need to ensure the teams are always aware of key things we are trying to learn from the world, and in turn be able to push this information to product management. Conversely, they need to be the voice of product management. 

Product management responsibilities need to be embedded in the DNA of all business functions. This reduces product management's responsibility to organizing incoming incoming information, look for the most compelling hypothesis and test them by devoting maximum development fire power at these hypothesis. In turn, product management leans on the business to evaluate the outcomes.

What does this mean for other functions?

For different stakeholders, I will outline key areas where they need to think like a product manager in addition to their core responsibilities.

Sales rep

• When a prospect ask about functionality that the product doesn't have, learn if the prospect would value the functionality they are seeking. If yes, what problem would they solve with it?
• When a prospect is unwilling to buy at proposed price, learn how much they are willing to pay for the product
• When a prospect is unwilling to buy, learn what problems the product would need to solve for them for them to value it

In turn, they should be equipped to talk about the kinds of problems we are looking to solve in the upcoming months, and learn if prospects value solutions to those problems. A false or a positive signal both unlock different kinds of investments based on product management's judgment.

Marketing

• Build new content and drive awareness based on what the product team is building
• Create and spread the word on success stories
• Build a compelling set of use cases for the product 

Customer success

• Learn of blockers to product adoption
• Recruit users for usability testing and user interviews
• Learn how latest features are being used & provide insight to product management

Once these expectations are clear, the organization needs to be instrumented such that information can flow freely. We intend to rely on a combination of Asana, Slack, & Totango to stay aligned here.

The benefits of such organization are immense. Due to inherent decentralization of product management, this can shorten cycle times significantly, and scale well as the business & product footprint grows without the need for hiring additional product managers, or impacting cycle times. This keeps our business responsive to customer needs and allows us to bridge the gap between problems and solutions rapidly.

Amazon's Last Mile Problem (Hint - It's not Amazon)

Amazon recently announced that it is considering a significant increase in annual subscription price for Amazon Prime. Amazon Prime, for those who live under a rock, gives its subscribers free 2-day shipping on almost all Amazon purchases. This price increase will net Amazon significant revenue. However, it is fair to assume that this will increase pressure on Amazon to deliver on its promise of 2 day deliver - something observers claim doesn't always happen today.

This brings me to what I see as Amazon's major challenge - the last mile of their delivery chain which is when the goods meet the shopper. Of the 2 occasions I've been disappointed by Amazon wasn't Amazon's fault.

On the first occasion, I wanted a package rerouted after it had made its way to the local fulfillment center. I called Amazon's customer service requesting a re-routing of my package. However, after 10-15 minutes of phone conversation with Amazon (in which time they basically told me nothing more than what I looked up by tracking my FedEx package), they simply referred me to Fedex saying they are unable to change delivery. I called FedEx, and as I expected, FedEx simply said my request was simply pointless.

On the second occassion, I ordered a pair of running shoes. I put 25-30 miles on my shoes every week, and few things give me as much joy. I was looking forward to getting my hands on Mizuno's Wave Rider 16 (that is one good looking pair of shoes!) like a little kid waiting for Christmas. I ordered them to be delivered at my office since it pretty much guaranteed that I would be there to receive it. I waited till 8:30p on Friday, only to receive a notification from FedEx that no one was available to receive the package. I was thoroughly disappointed.

What is interesting about the two events was that Amazon wasn't to blame, it was FedEx. Yet, all my disappointment was directed towards Amazon.

As a shopper, I want my goods faster & cheaper than ever before. Jeff Bezos gets it. I am certain that Bezos gets the sheer joy shoppers get from getting their hands on things they want, and I am certain he doesn't like being vulnerable to courier delivery services that culturally not as in tune with the customer as Amazon is, nor are they as motivated to improve as Amazon is.

How Amazon addresses these challenges will be interesting to watch, especially in light on increased subscription for Prime. I will continue to pay... for now.

Importance of free will for employee engagement

Like many other companies, at Sefaira we realize that empowering and engaging our entire team with the company's mission is critical for our success. This implies those closest to the front lines, i.e. our customers and our code, should feel complete engagement with the goal of the company & motivated to make decisions and act in the interest of Sefaira's success.

The business value of employee engagement isn't suspect, and can be proven by empirical data. Anecdotally, we feel this helps with improve employee happiness as well. However, I recently read a study by Mihaly Csikszentmihalyi‎ which provides empirical data in favor of this.

Using his novel Experience Sampling Method, Csikszentmihalyi‎ studied a population of workers across a wide spectrum to understand when people enjoyed themselves. He learnt the following -

• While at work, 54% of the population said they experience enjoyment. During this state of enjoyment they felt challenged & they felt their skills were being used well
• While at home or during leisure, only 18% of the population said they experience enjoyment. Interestingly, a majority of workers felt apathy & boredom during leisure hours

This clearly showed that people enjoy themselves more at work than they do during leisure hours. However, a 2nd question showed an interesting paradox. A majority of workers who said they were enjoying themselves at work said they'd rather be indulging in a leisure activity. Also, a majority of workers who were bored during leisure stated that they'd rather not be doing something else. 

This means that a majority of workers would rather be bored & apathetic, than engaged in enjoyable work. This was a paradoxical finding. The author states that this can likely be attributed to the social perception that the nature of work is not determined by employees (even though work itself is challenging & enjoyable), and is not under the employee's free will. This lack of free will leads employees to choose to spend their time being apathetic, away from work they truly enjoy.

For us as managers, the lesson is clear. The more employees feel like they are acting under free will, the happier they will be, as they will seek the enjoyment that comes from work. Of course, the business benefits from this if it is organized to allow employees to decide & act in the interest of business.

Two ways in which quotas can debilitate startups

Quotas(1) play a very important role in businesses. They are the chief instrument for translating business objectives into measurable milestones. Further, they translate measurable business goals into objectives for revenue generating functions in the business. Ultimately, they connect all other functions ranging from product development to marketing with overarching business objectives.

However, they suffer from two major weaknesses especially for startups who tie themselves to quotas like established businesses would.

Quotas are not actionable metrics

I was first introduced to the idea of actionable metrics while reading "Out of the Crisis" by Edwards Deming who used accident statistics as an example of an outcome-based metric that measures the result. However, it fails to provide any insight into what actions can be taken to improve the metric. Similarly, quotas are an outcome-based metric, which can't translate into actions as highlighted by Jason in this HBR article. Naturally, as Tom Tunguz points out in his follow up, they lead to anxiety in absence of actionable metrics (which are often process indicators) as they leave sales people powerless (2). Actionable metrics are important for sales as they result in constancy of purpose, and break business objectives down into fundamental metrics that can be acted on. In essence, actionable metrics translate sales into a machine that looks for continuous improvement as opposed to attainment.

Quotas can obfuscate reflection

Established businesses have a business model that works, they have a sense of market size and their customers, and therefore - business goals well informed. Startups are unlike established businesses. Business goals aren't informed by an objective market size, as very often startups are creating new markets based on their offerings, or fulfilling needs that the market didn't realize it had. This implies that unlike established businesses, startups are going through constant learning. A natural extension here is that startups  need to reflect about their sales processes independent of attaining quotas - way more than established businesses do. The challenge here is that quota attainment (or meeting budget) can sometimes result in inadequate reflection, and delay in actions which can debilitate a startup.

A question for more informed folks, especially those that sit on startup boards - Does measurement of leading indicators lead to better "retrospectives" by executive teams than meeting budget?

1 - I use quotas interchangeable with budgets here. I find that the difference between the two is essentially their probability weights. Budgets have 90% likelihood of being met, where as quotas might have 50%.

2 - In the face of goals, being powerless causes anxiety - a thesis proven by Mihaly Csikszentmihalyi in Flow

How we reduced our time-to-production by 90%, and quadrupled our velocity

Towards the middle of Q2 2013, I took over the responsibility of leading our product(s) along with Dave who took on the responsibility of managing our development team. For a development team of less than 10 developers, the challenge ahead of us was very clear

• We were doing ~ 5-10 releases a month across our product stack
• Our average elapsed time from conceptualizing a feature or improvement to delivery in production was ~ 100 days

Our mandate was to deliver value to the users faster by accelerating the rate of releases, and drive down time to production as much as we could. Essentially, we wanted to become more responsive to the needs of our customer & the business

The Old World

While there were many areas of improvement, I list what we noticed as top characteristics of our development process

• Developing user stories, use cases & scenarios - Our product management served as strong proxy for the customer & diligently detailed user stories & use cases to help the developers learn of our customers’ needs. This was the reasonable thing to do as our product requires significant input from domain experts, and prepping use cases to help developers understand the problems our customers want to solve.
• Strong competence in specific areas of our stack - Our developers had cultivated significant competence across clearly identified areas of our development stack. Again, like many teams, we matched skills to areas of code. This allowed us to move the respective areas of code faster & faster over time
• Quality assurance by QA team & product management before shipping - We had developed a culture of quality assurance as the last gate of development before shipping to the customer. This allowed us to catch areas where we hadn’t met our customers needs, or the user story wasn’t completely satisfied before shipping the product. This is difficult to argue against.

As strange as it may sound, these were the very practices that held us back. Let’s look at them up close

• Developing user stories, use cases & scenarios – On face, this is a very good idea. However, this abstracted our developers from the customers which made it difficult for our developers to understand the pains of our customers. The answer here was clear. We needed to move our product team’s center of gravity closer to the customer, and eliminate product management as a bottleneck. Instead, product management would serve as a bridge to the customer, not a gatekeeper.
• Strong competence in specific areas of our stack – This worked very well for us in the early days. However, given that everything we ship is cross functional, we needed developers from every area of code to ship anything. This led to key man dependencies that bottlenecked our development efforts.
• Quality assurance by QA team & product management before shipping – Since product management had the best understanding of our users’ problems, this was considered an important & critical step. Product management would request domain experts on our team to validate the product against expected behavior prior to shipping. This led to significant time added after development & prior to shipping.

The New World

We developed principles that we would follow to break away from this practice & reduce our time to production while increasing our velocity (as measured by # of releases)

• Move our center of gravity closer to our customer – The purpose here was singular. We needed our development team to have a visceral understanding of our customer’s goals & pains. We did this by holding multiple seminars on architect’s workflow as well as building physics in-house. In parallel, we started conducting user interviews & usability sessions – the caveat being that the latter would be led by our developers, coordinated by product management
• Reduce batch size – We decided that we would work relentlessly to minimize the quantum of product enhancement that can be meaningfully sent to our customers. We also made commensurate moves across our technical stack to facilitate this
• Decentralize releases – We wanted developers to be responsible for shipping the product, right from the stage of conceptualizing the ticket to release. This could only be done if we reduced the size of each release, product wise, to something that could be managed by a developer. Further, it required our services to move independently of each other when needed.
• Enhance code coverage – We decided to make conscious effort, even at the cost of near term benefit, of sharing code across the team. This was done to allow multiple developers to move code across the stack.

The result, thus far

Time to production vs. creation date

• We drove down our time to production (defined as time from conceptualization to production) from 100 days on average to 10 days. As a result, we are now very responsive both to our customers' needs and our vision for the product.

# of releases per month

• We have nearly quadrupled the number of releases we do. This really paid off for us in Q4 where we launched 2 new products – Sefaira for SketchUp & Sefaira for Revit, and did multiple releases which benefited our customers.
• In terms of code coverage, 80% of our developers contributed to areas of our code where we invested 80% of development budget

While many challenges lie ahead of us in terms of driving growth, it is encouraging to note that we are accelerating our product development. Our customers & investors expect nothing less.

The Case for Gated Releases

At Sefaira, we measure the performance of our product team based on frequency of releases, following the message well captured in this article. I am always making the case for releasing frequently to the point where this is now captured on our meme wall. However, there exists a real conflict between the frequency of releases & the quality of the product that goes out of the door. The tension is fairly predictable - the less frequently we release, the more time we have to improve the quality of the product that goes out (implying better performance, more feature completeness, more polish & less bugs). This tension is real, and troubles many startups including ours

The argument goes like this. If you send out a product with sub-optimal performance, low polish, known areas of improvement, & some undiscovered bugs to the user, you risk getting written off as the user will be disappointed. However, if you spend a significant time improving the product performance, by covering more use cases, adding polish & finding/squashing bugs before the customer gets it, you can ensure that the user will not be disappointed by the product. Who in the right frame of their mind would argue against such sound "user focused" behavior by the product team?

Well, I will and here are reasons why the argument above doesn't apply to most startups (i.e. companies seeking high growth as defined by Paul Graham). There are many unknowns at a startup.

You don't know what your customers want

While you might think you know what your customers want, you have absolutely no knowledge of how much value your users attribute to new features or products. All startups are constantly trying to discover the next product & set of features/functionality/performance improvement which will fuel their growth. This is distinct from optimizing an existing product for your mainstream customers when their desires are absolutely clear. Apple knows its users want higher battery life, and Amazon knows their users want products at a better price, faster (*). On the other hand, Google does not know what its users want from Google Glass, Facebook doesn't know how much its users engage with in-feed video ads (**).

You can't justify product investment based on projections of value delivered

Absent of a quantifiable basis for value of new features, you don't know how much spend can be justified for development of these new features. This leads to significant risk of wasting precious capital on features that don't deliver value to the users & business -  capital that should be used with significant prudence. Note that I say prudence, not sparingly. Sometimes this is misconstrued & product teams end up solving the easy problem that don't require high investment. At Sefaira, when we were were doing early usability tests of real time analysis plugin, we didn't connect the plugin with our cloud based engines. However, we learnt quickly that this could undercut the viability of our product. At this time, we invested time & money in connecting the new plugin with our cloud based engines. Note that the product still lacked polish, which brings me to the last point.

You can't polish your way to success

While polish can amplify engagement of a useful product, polish doesn't convert a useless product into an engaging one. Products without polish are promising at worst, and highly engaging at best. Google Glass is promising, whereas Facebook's early version showed very high engagement. If you compare Mailbox with Boomerang, you see a clear case where polish clearly creates a highly engaging experience with Mailbox whereas poor design with Boomerang keeps its squarely in the "promising" bracket compared to Mailbox.

While the above holds true, this is still not a case for shipping a product that has known areas of improvement, known & unknown bugs & poor polish. Users can get turned off with a poor product. At Sefaira, we surveyed our early users who either did not engage with the product, or didn't come back to use it after 1st experience. Nearly half of these users attributed their lack of engagement to product deficiencies, and almost all of them engaged with the product after our outreach which listed product improvements since their first experience. The key to re-engagement is rapid iteration to add value to users, and ability to reach out to these users with clearly identified improvements. It should be noted that a significant majority of users remained engaged or increased their engagement while we continued to add new users, and more features & performance - an important trait of gated releases.

This brings me to the case for gated releases. I define gated releases, as the roll out of new functionality or product in waves to the entire user base, with well identified metrics that need to be met before the next "gate" is opened & users let in.

Release for learning

Gated releases allow you to learn based on early user experience, and enable meaningful iteration on the product itself. This iteration is based on feedback from a prior batch of users, and unlocks budget for the next gate. Metrics that need to improve are identified & measured. It is important to identify the size of each batch of users such that you can learn meaningfully from it, and ensure that your product team is capable of gathering feedback without getting overwhelmed.

Releases become forgiving

All product teams should take quality assurance seriously. However, there is such a thing as too much testing which can be quantified when you are testing aspects of the product that you are not looking to measure. These aspects are discoverable by you & your users, but simply don't matter for what you are trying to learn. If you roll out such a product through a "gate", you will quickly discover these aspects of your product & you will learn how much it held back user experience. Through the next iteration, you can fix these issues before opening the next "gate". Further, frequent releases allow you to fix mistakes quickly as you can quickly identify source of those mistakes due to the "small delta" nature of gated releases.

Releases unlock budget for next iteration

For all startup product teams, product development should be a series of carefully placed bets. You need to understand what bets to place by sampling your user base as often as possible. Gated releases allow for rapid & meaningful sampling of your user base

Releases eliminate intrigue & win support

A product team that doesn't ship, as well meaning as their reasons might be, can lose stakeholder & customer patience. If you are building it, ship it because shipping delivers value. However, a poor product can kill you. To avoid that fate, roll it out through "gates" of validation

To summarize, our experience has suggested that gated releases provide validated learning and a basis for product investment. It is extremely useful for startups to encourage the practice of gated releases, and it comes with some important pre-requisites

• Infrastructure that enables gated releases
• Development practice that buys into moving fast with focus on value
• Communication to the business that ensures everyone is informed & gathering feedback

* - Amazon still doesn't know how much premium customers attribute to faster shipping, and constantly experiments to determine this. Apple on the other hand can be fairly sure that there is no premium for extra battery life. 

** - Facebook likely knows that its customers attribute significant value to video ads, though only if its users see them.

One use case that validated my decision to switch to Android

Like many, I became an iPhone user many moons ago because I didn't want to miss out on a device revolution. After being an iPhone user for 5+ years, I had a sneaking suspicion that I was missing out on a similar revolution - one that was blurring the lines between the device & the internet. So I switched to an Android phone (the device doesn't matter, so it shall stay anonymous)

Sure, Android does some things that I can best describe as ranging from creepy (automatically processing images through Goggles) to nifty (Google Now gives me timely information on weather, local events, etc). However, one use case validated my decision to switch to Android.

I use Open Table for dinner reservations. Back in the iPhone days, I would make dinner reservations, and then set up a calendar invite with the reservation. I relied on my habit of looking at the calendar every few hours to remember that I had a dinner reservation. I would check how far the restaurant was, plan the route & figure out how far ahead i needed to leave to get to dinner on time. Of course, as I was pulling into the restaurant, I would get a completely worthless reminder from my calendar.

This is what the same use case looks like with Android. I make a dinner reservation on Open Table. Since Google scans my email, it knows that I have a reservation at a restaurant on a given date & time, so instead of adding it to the calendar, it simply shows up on Google Now on the day of reservation. Further, it knows where the reservation is, so it reminds me to leave for dinner based on the commute time to the restaurant including a route plan. Finally, I don't get an annoying reminder 10 minutes before I pull into the restaurant.

This blurring of lines between the device & the internet is why I think Android will eat iOS. I tell people that Android taps into a greater ecosystem, and I get quizzical looks as people point me to the iOS app store which I construe as a very narrow definition of ecosystem. Yes, iOS has an amazing app store and Android is staying close. However, when it comes to tapping into the internet is concerned, iOS isn't even close.