Have you ever worried about knowing what to do next for your product? Often, the qualitative and quantitative data you have on hand about your product can provide only a fuzzy outline of what is happening and it can be challenging to know if you’re heading in the right direction.
While product managers have a variety of techniques to cope with this ambiguity, from the Mind the Product slack channel to thunderdome-like meetings, one powerful solution is product review. This is a formal review of the product’s progress by senior product managers in order to assess the status and suggest avenues to explore. I decided to use this tool when I was the VP of product management and innovation at Telenor, Norway’s state-owned telecommunications provider.
Why Product Review?
Telenor has limited resources to spend on developing digital products. Product review is a way to assess how the products were doing, and adjust the resources allocated to the products. Resources in this case meant people, such as assigning a new designer, and money, for hiring an advertising agency.
How did you Identify Product Stages?
Telenor has a gate system to categorize products ranging from ideas (Stars) through to mature products (Space Stations). Each product is assessed by its level of completion of the requirements for each stage of development, and this assessment translates into the categorization of the product. Once the product is assigned to a designated category, it goes through the product review for that stage.
Stars, Bottle Rocket, and Satellite are considered early-stage products and have metrics related to learning. Space Shuttle and Space Station are considered mature products and have metrics related to financial earnings.
Running specific product reviews for specific stages of development is an important component of Telenor’s product review. There is no point in asking a team that’s immersed in testing basic assumptions about a product how they plan to scale the solution technically – it’s an irrelevant question for the team. Previous to Telenor’s product review program, all products had been assessed on the same criteria, which didn’t align with the reality of the different stages of development.
Frequency of Product Reviews
In general, product reviews are run monthly for Stars and Bottle Rockets, and quarterly for Satellites, Space Shuttles and Space Station. The product manager and reviewing senior product resources would agree on a date for the product review meeting, allowing time for the product manager to prepare. The earliest-stage products have fewer resources and are more likely to produce learnings in a short period of time that will alter their direction, hence the need for a more frequent product reviews.
What Happens During Product Review Meetings?
Product review is an opportunity for product managers to present the issues they encounter to senior product resources. Discussion of these issues may lead to a reallocation of resources – ranging from help in hiring a design resource to a few hours with the legal team. The manager who controls the budget for the product team is also typically present, allowing for direct requests to be made for changes. The senior product resources are then responsible for securing the assistance requested as part of the product review.
Product metrics are also discussed. For the first two phases of product (Stars and Bottle Rocket at Telenor), metrics evolve as we learn more about the problem being investigated. For the third stage (Satellite), metrics remain relatively stable over two or three product review cycles (every six to nine months), and change only as significant, new information comes to light (for example, a technical architecture change needed for progressing to new level of scale.) For the last two phases (Space Shuttle and Space Station), metrics do not change (at Telenor the timeframe is 18 months) as the problem is already well understood, the proposed solution is far along in development, and the markets they address are well-defined. It is possible that the metrics for Space Shuttle and Space Station could change over a longer period of time.
After our first product review, we realized that the use of templates would make it easier for teams to organize the data. While it isn’t necessary to adhere strictly to the template, the questions posed in the template need to be answered. We found we rewrote the templates several times over the course of a year to better incorporate changes in the product review process as we reviewed more products. For example, we added an “onboarding” template that had a short set of questions from each of the stages so that we could better assess where products were in the maturity scale.
What’s the Most Effective Environment for Product Review?
Product reviews are most helpful for organizations with junior and mid-level product managers who don’t have senior product managers as their line managers. Product review is a way to make the most of scarce senior product manager resources, and can work even when the senior product managers are brought in as consultants (as was done with two of the senior product manager resources we used at Telenor). Having said that, I believe that all levels of product managers would benefit from taking the time to distill what they have learned in the preceding months and to talk through their next steps with another product manager resource. As senior product managers tend to seek out these learnings independently, the benefits may simply be less dramatic than for an organization with less seniority in their product management department.
In addition, the organization should support the idea of dynamically allocating resources to support further product development. Rigid budget structures for product development spend can cause problems when a step-change in resources is needed to take advantage of the results of the product development process. Buy-in to the idea of changing direction as new information is gathered is critical for line managers who control the budgets for product development.
Lastly, as with all systems, product review also requires senior management support. Support is needed in areas such as budgeting, allowing for space for promising products to receive additional funding, or human resource policies, allowing employees to fluidly move between teams to build their careers.
How Would you Improve Future Product Review Systems?
Our team learned a lot about the challenges of deploying a product review system across a large organization like Telenor. Here are a few ideas about what we would keep in mind for next time:
- Start with a template. We tested both with and without a template, we found that employing a template was a much better way to get across the kinds of topics that could be addressed in product review.
- Set up a system for regular feedback. Multiple countries, multiple cultures, multiple products – there was a distinct need for setting up a regular feedback mechanism to assess the usefulness of the product review system.
- Face-to-face discussions work best for detailed discussions. Video conference calls do not work as well as face-to-face for establishing rapport between teams. We had to conduct some product reviews on video or audio calls, and these were not considered as helpful by the teams engaged.
