Building better products through balanced inquiry

Like many other product companies, we have a relentless focus on moving our product team's center of gravity outside the building. We do so by doing frequent usability testing, and are currently maintaining 1 session/week. 

A challenge we often grapple with, when it comes to usability tests, is the conflict between structuring our questions which allows contextual inquiry, vs. a free flowing nature of conversation which allows us to discover areas we were blind to. Our view is that both of these modes are important, and must co-exist. Here is why.

Benefits of structured inquiry

Structured inquiry allows product teams to validate the product vision. These are often answers to problems that the user hasn't discovered yet or didn't consider possible. This is representative of the famous quote (allegedly) from Henry Ford -  "If I had asked people what they wanted, they would have said faster horses", to suggest that people couldn't imagine that they needed a car. Improvement of the product along this line leads to disruption in user work flow, & adoption often by the "irrational" user (due to novelty of the product). For us at Sefaira, this included:

• Real-time analysis - Our users knew that building performance analysis takes several minutes to hours, that they would've never expected to see results in seconds. This changes their work flow & behavior in ways they didn't expect
• Feedback as you design - Our users were used to performing energy analysis in separate specialized analysis software, and didn't consider integration with their design environments as a possibility
• Results first paradigm - All current analysis software forces users to follow a workflow of specifying all inputs & then examining results - a classic design to performance paradigm. Users couldn't imagine that a paradigm which flipped this (performance to design) could even exist.

It would be fair to state that this helps us discover "global optimum"

Benefits of free flowing inquiry

Free flowing inquiry allows product teams to uncover their blind spots. These are often discoveries about the user's patterns & work flows that the product team was blind to, or had assumed during product development. Improvement of the product along this line leads to greater adoption by the "mainstream" or "rational" user. For us at Sefaira, this led us to discover the importance of many things we were ignorant of

• Widespread use of collections & layers in SketchUp
• Widespread (mis)use of custom families in Revit
• Arduous nature of information management in building design software

This helps us discover "local optimum".

Conclusion

Balanced inquiry during usability testing leads to realization of product vision while ensuring deeper adoption of the product by the mainstream user. Product teams can balance inquiry by doing the following

• Note down 3-5 things we are trying to learn about the product
• Give users a lot of white space to talk freely, and echo their questions back to them
• Ask them what they would build next in the product, & why

I encourage product teams to identify benefits their users have realized & features built in their product, and correlate (as best as possible) those improvements to engagement & revenue metrics of the product. This presents a meaningful view of the trajectory of your product & its impact on business.