post

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Successful product management in companies where it’s least understood

Measuring critical user journeys – Javier Andrés Bargas-Avila (UX Director, Google)

A new way to look at applied experiments

Product survey findings: Only 15% of users are embracing AI features 

Building high-performing product teams: The PDLC approach

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Guest Post

Overcoming the 20% feature usage challenge

Product Management

Three product leaders to learn from at #mtpcon inside Pendomonium

Jacob de Lichtenberg is Chief Product Officer at Forecast with a background in Citrix, CSC, and Trustpilot. He is obsessed with empowering teams to build products that renew the world as we know it.

Jacob de Lichtenberg

<p>How do you decide what to prioritize or whether to believe what customers and prospects are telling you about your product? Here, I&#8217;ll explain how applying the hierarchy of value certainty, shown below, can help you to answer these questions and make your product profitable.</p>
<p><img decoding="async" src="//wp-content/uploads/2019/12/bccc105c-97bc-45e1-80ed-501a8dbf6c19_image4.png" alt="" /></p>
<p>A core skill of product management is <span style="font-style: italic;">prioritization</span>. For most, the end goal is to prioritize the things that will make you become the most profitable company. At the heart of your prioritization skills should lie <span style="font-style: italic;">value</span> and <span style="font-style: italic;">effort. </span>In B2B companies, value should ultimately generate profit (get new customers, upsell to current customers, or avoid churn), and it ends up in an equation like this.</p>
<p><img loading="lazy" decoding="async" class="wp-image-18128 aligncenter" src="//wp-content/uploads/2019/12/most-profit.png" alt="" width="450" height="93" /></p>
<p>&nbsp;</p>
<p>However, not all methods are equally effective when deciding what is valuable. In fact, some methods are pretty poor. Unfortunately, it is these methods that product managers often use.</p>
<h2>Your Skills Resume: Can you Identify Profit?</h2>
<p>Some product managers make poor decisions because they don’t have the right skills. They identify a feature as profitable, when it isn’t. One problem is that they believe people too easily. I have a background in behavioral psychology, and one lesson I&#8217;ve learned from this field is that people don’t always know why they feel like they do[1], and they might rationalize why they say something and come up with an explanation, even though they don’t really know why[2].</p>
<p>Take this experiment[1], where college students completed questionnaires about their mood and other data points every day for five weeks. Researchers then asked the students what they believed had influenced their mood. Since they had the data, they could now compare the students’ subjective estimations with the verified correlations between data and mood. Look at these results:</p>
<table>
<tbody>
<tr>
<td><span style="font-weight: bold;">Predictor variable</span></td>
<td><span style="font-weight: bold;">Subjective<br />
estimate</span></td>
<td><span style="font-weight: bold;">Objective correlation</span></td>
</tr>
<tr>
<td>Sleep</td>
<td>0.33</td>
<td>0.04</td>
</tr>
<tr>
<td>Friday</td>
<td>0.60</td>
<td>0.05</td>
</tr>
<tr>
<td>Saturday</td>
<td>0.59</td>
<td>0.13</td>
</tr>
<tr>
<td>Weather</td>
<td>0.49</td>
<td>0.27</td>
</tr>
<tr>
<td>Health</td>
<td>0.51</td>
<td>0.51</td>
</tr>
<tr>
<td>Relationships</td>
<td>0.59</td>
<td>0.50</td>
</tr>
<tr>
<td>Food</td>
<td>0.16</td>
<td>0.35</td>
</tr>
</tbody>
</table>
<p>&nbsp;</p>
<p>Notice that people believe the <span style="font-style: italic;">day of the week</span> has a massive influence. It doesn’t. And they significantly underestimate the influence of <span style="font-style: italic;">food</span>. But they do get some things right: <span style="font-style: italic;">health and relationships</span>.</p>
<p>The analogy to product management is that customers don’t always know what they want. And if you show them something that doesn’t exist, they often come up with an estimate similar to this experiment. The point is, what customers say is not enough to create certainty that something is valuable and will be profitable. The reality about why product managers fail is often related to believing what customers say.</p>
<h2>How Something Seems Valuable, but it isn’t</h2>
<p>When I worked at Trustpilot, we had solid evidence that consumers wanted to read reviews about products. (Currently, Trustpilot only shows company reviews). Since Trustpilot receives revenue from companies paying to collect, analyze, and promote their reviews (it’s also free with fewer features), it was important that companies found product reviews valuable to display as well. Therefore, we scheduled 20 interviews.</p>
<p>The feedback on the concept was stunning: 15 out of 20 companies were excited about the idea.</p>
<h4><span style="font-weight: bold;">75% loved the idea</span></h4>
<p>We asked them to confirm their interest and participate in a beta release. These are some of the confirmation emails we got:</p>
<table>
<tbody>
<tr>
<td><span style="font-style: italic;">“Thank you for today&#8217;s meeting and presentation and I confirm our interest in both of the concepts being developed and look forward to hearing more and participating in beta trials.”</span></td>
<td><span style="font-style: italic;">“I think it is a very good idea. The number one reason is to help us make sure that we have good customer experiences. Actually, … this adds more customer confidence in us and our products.”</span></td>
<td><span style="font-style: italic;">“We have 7000 customers … Many of them also leave a product review on Trustpilot, which we wish would be shown publicly on Trustpilot and not only on our own website… We think this is awesome.“</span></td>
</tr>
</tbody>
</table>
<p>&nbsp;</p>
<p>Imagine this was your company. Doesn’t it sound like a very valuable thing to prioritize if 75% of customers want it? You might be inclined to say, let&#8217;s do it! But the product manager answer is: You don’t know!</p>
<h2>The Hierarchy of Value Certainty</h2>
<h3>Say it</h3>
<p><img decoding="async" src="//wp-content/uploads/2019/12/22da1d26-6d16-46da-857d-ddd5f9202757_image5.png" alt="" /></p>
<p>In the hierarchy I’m suggesting, if a customer tells you something is valuable, it gives you the lowest level of certainty that it will be valuable to the customer. You need more. The Trustpilot case showed 75% of customers liked our idea. But it turned out that this information was useless as soon as we did something more, something I&#8217;ll return to later in this article.</p>
<h3>Un-shame… and Wait for Endorsement</h3>
<p><img decoding="async" src="//wp-content/uploads/2019/12/092fa72f-0317-4ae2-ab31-e1d5b53e732c_image3.png" alt="" /></p>
<p>If you interview your customers about a specific feature, rather than ask what they think about an idea or concept – lead them in the opposite direction. Most customers want an interview to be comfortable. Many customers will try to answer in a way to make the interview pleasurable.  So you need to lead them away from what they think you&#8217;ll like. Here is an example:</p>
<p><span style="font-style: italic;">Hi. My name is Jacob. I am going to show you a mockup of an idea the company is working on. It’s from another team, so you can be completely honest with me. I have just been tasked to research the idea. To be honest, I am not sure I fully understand it, but this is not my area…</span></p>
<p>When you show the mockups,  start shaming the concept:</p>
<p><span style="font-style: italic;">It just doesn’t seem like it’s their best work / best idea.</span></p>
<p>The point here is that the customer should start defending the idea. If they don’t, it’s probably not as valuable as you think.</p>
<h3>Prioritize Over Other Valuable Features</h3>
<p><img decoding="async" src="//wp-content/uploads/2019/12/238d0290-9ca5-4e42-95eb-c20f781bcc78_image6.png" alt="" /></p>
<p>An even better way to understand the value of an idea is to compare it to other things. If you are planning to build a Chrome extension to your product, it’s easy to ask customers to prioritize the idea over other things during interviews:</p>
<table>
<tbody>
<tr>
<td>Question: <span style="font-style: italic;">Out of the following, what do you think would be most important for us to work on: </span></p>
<ul>
<li><span style="font-style: italic;">A mobile app</span></li>
<li><span style="font-style: italic;">A Chrome extension </span>(this is the thing you are really asking about)</li>
<li><span style="font-style: italic;">A desktop application</span></li>
</ul>
<p>(Or you could ask the customer to rank them)</td>
</tr>
</tbody>
</table>
<p>&nbsp;</p>
<p>While value/effort  might be bad for some of the other suggestions, you should consider if the idea in focus is valuable enough if customers don’t rank it on top. Remember that real life is often messy, and asking for prioritization will often create messy results in contrast to the beautifully simple result of <span style="font-style: italic;">“75% of our customers love this idea”</span> you get if you ask them. Asking customers for feedback can provide a lot of insight. But it <span style="font-weight: bold;">does not provide validation</span>.</p>
<h3>Pretend it’s Real</h3>
<p><img decoding="async" src="//wp-content/uploads/2019/12/846e61cd-bd82-4a7b-ae51-ed93cfde7429_image8.png" alt="" /></p>
<p>Pretending that a feature exists is a great way to get validation. Advocated by people like Alberto Savoia in <a href="https://www.pretotyping.org" target="”_blank”" rel="”noopener” noopener noreferrer">Pretotyping</a> and Eric Ries in <a href="http://theleanstartup.com/" target="”_blank”" rel="”noopener” noopener noreferrer">The Lean Startup</a>, these are good methods to test if something is valuable fast, and it is a product management skill you should learn.</p>
<p><img decoding="async" src="//wp-content/uploads/2019/12/8d6cbb8c-e99a-4e2e-a937-ee8b36f45d74_image12.png" alt="" /></p>
<p>Let me give you an example from my days at Citrix. We looked at using Citrix’s current feature for new products, and one idea was to build an Uber-clone, an IT support site, where consumers could go to get help from multiple freelance IT support specialists. They would get instant help, so the working title was “Instant Relief”, and we even bought the domain Instant-relief.com.</p>
<p>As in the Trustpilot example, we set up 25 user interviews with IT freelance support specialists and consumers to get their feedback. The results were astonishing: 75% of consumers said they would use and pay for the service, and 78% of IT support specialists would sign up. But because what people say doesn’t really validate an idea, we wanted more certainty.</p>
<p>Instead of building anything, we simply mocked up a web page – a user interface that looked like a working page, but whoever you clicked to get support from, you connected to the same internal support specialist through a chat session, where we would pull people into Citrix’s own systems. We could get traffic to this unknown site by buying it through AdWords. This concept is called a <a href="https://en.wikipedia.org/wiki/The_Turk" target="”_blank”" rel="”noopener” noopener noreferrer">mechanical turk</a>.</p>
<p>As you may have guessed, Instant-relief.com doesn’t exist today. Cases like these demonstrate that when people believe something exists, their behavior is very different from what they say.</p>
<p>There are other methods for pretending something exists (like having a sales person trying to sell your idea even though it doesn’t exist). Many variations of <span style="font-style: italic;">pretending it’s real</span> will let you measure behavior, which is the next level of the value-certainty hierarchy.</p>
<h3>Behavior</h3>
<p><img decoding="async" src="//wp-content/uploads/2019/12/116ad791-a4fc-46fb-8f4b-d0d77ae75468_image7.png" alt="" /></p>
<p>There are many great ways of using behavioral data to know if something will be valuable.</p>
<p><span style="font-weight: bold;">Tracking</span>: At <a href="https://hubs.ly/H0lrhQg0" target="”_blank”" rel="”noopener” noopener noreferrer">Forecast</a>, we want to change the future of project management, so that projects succeed and become profitable. While our product combines resource and project management functionality with artificial intelligence, a big part of project success relies on teamwork. We had an idea that the page for scoping out the project (meant for project managers), would actually help team members while executing the project. As we have tracking and customer data readily available <span style="display: inline !important; float: none; background-color: #ffffff; color: #3f3f3f; cursor: text; font-family: 'Merriweather',serif; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px;">at Forecast</span>, we could quickly run a test:</p>
<figure style="width: 936px" class="wp-caption alignnone"><img loading="lazy" decoding="async" src="//wp-content/uploads/2019/12/d21b2aa8-619d-48b8-935b-e1b94221a90f_image11.png" alt="" width="936" height="736" /><figcaption class="wp-caption-text">Forecast ran a test by making a page accessible to a sample of users</figcaption></figure>
<p>Instead of asking team members if they would use this type of page, we simply selected a sample of customers and made the page accessible to them. We then tracked behavior on the page to see if the sample users were using the page after three weeks.</p>
<p>Tracking data only tells you about your current customers. Sometimes there might be other ways of working – but you might not be able to see this in the data.</p>
<p>There are other methods for looking at behaviour, Lost deals for example. If customers start engaging with your sales department and love your product, but end up not buying because you lack <span style="font-style: italic;">something</span>, then there is behavioral proof that this <span style="font-style: italic;">something</span> is very valuable to them. Experience it a few times, and you probably have proof that this <span style="font-style: italic;">something</span> is something you should research.</p>
<p>That requires discipline from your sales department, which many startups do not have time for &#8211; though at a recent conference I attended every successful startup CEO said &#8220;if you are not disciplined in documenting the sales process, you will never succeed&#8221;.</p>
<h3>Investment</h3>
<p><img decoding="async" src="//wp-content/uploads/2019/12/4526ea2a-ce8d-4b20-baa9-8f8a3c03a143_image10.png" alt="" /></p>
<figure style="width: 914px" class="wp-caption alignnone"><img loading="lazy" decoding="async" src="//wp-content/uploads/2019/12/6f28e1ef-3d6e-4fef-b5a1-c40285388d34_image13.png" alt="" width="914" height="918" /><figcaption class="wp-caption-text">Trustpilot asked customers to invest by sending their product catalogs</figcaption></figure>
<p>If customers are interested enough in the new idea, they are willing to work for it. So if you can get them to invest, you are more certain that is something valuable to them.</p>
<p>Let’s return to the Trustpilot case. Remember that 75% of customers said they loved the idea.</p>
<p>We then asked the same customers to <span style="font-style: italic;">invest</span> by sending us their product catalog. This is not a trivial task. It requires significant work from their side, and might even include others in their organization.</p>
<p>So how many sent us their product catalog? 3 out of 20.</p>
<h4><span style="font-weight: bold;">15% invested</span></h4>
<p>For me, this is the real number you should look at. Instead of getting overwhelmed by what people say, you should look at how many people actually invest or commit to the idea. If you are working up a business case, this is the number you should use.</p>
<p>Thinking about how to get people to <span style="font-style: italic;">invest </span>is definitely a skill to keep practising.</p>
<h3>Commitment</h3>
<p><img decoding="async" src="//wp-content/uploads/2019/12/cf09e383-e85b-4688-97a6-f6e53f9160ce_image9.png" alt="" /></p>
<p>If there is <span style="font-style: italic;">one thing</span> you need in order to know if something is valuable and will make your product profitable, then it is customer commitment in the form of a signed contract. That is hard to get, because few companies are willing to commit to something that doesn’t exist. But if people are really excited about something, they are also willing to commit to get it.</p>
<p>Before Ross Mason started Mulesoft (Valuation at IPO: $2.9 billion), the investors were able to call JP Morgan as it was ready to buy the product, if there was a company to buy from. This is an example of a potential customer being fully committed.</p>
<p>You should always try to get commitment! The problem is that many product managers never ask for commitment, because it can feel uncomfortable – mainly because they see it as changing the conversion from research/discovery to sales. I highly recommend moving conversations in this direction, especially when it comes to entirely new features in your product. These are the riskiest investments, and this is where you need the highest certainty.</p>
<h2>Value: The Core of Prioritization</h2>
<p>Part of your skillset as a product manager should be to know about prioritization frameworks: All of these have <span style="font-style: italic;">value</span> as a core of how to prioritize. It may be known as satisfaction, impact, value, and even revenue. One of the best-known techniques is <a href="https://www.intercom.com/blog/rice-simple-prioritization-for-product-managers/" target="”_blank”" rel="”noopener” noopener noreferrer">RICE prioritization</a> from Intercom:</p>
<p><img loading="lazy" decoding="async" class=" wp-image-18150 aligncenter" src="//wp-content/uploads/2019/12/rice.png" alt="" width="339" height="69" /></p>
<p>&nbsp;</p>
<p>Using my suggested hierarchy helps you with both <span style="font-style: italic;">impact</span> and <span style="font-style: italic;">confidence</span>. The higher you are in the hierarchy of value certainty, the more <span style="font-style: italic;">confidence </span>you eventually have. This is the main use. (Remember certainty about effort and reach). Evaluating the <span style="font-style: italic;">impact</span> (which Intercom’s Sean McBride handles cleverly, but also loosely, by putting in a number between 1-3) is much easier if you are certain about the value. By using the methods higher in the hierarchy, you basically prove the <span style="font-style: italic;">value/impact</span>.</p>
<h2>Your Skill: Ask About the Hierarchy Level</h2>
<p>When people say something is valuable, always ask on what level of the hierarchy there is evidence of its value. This is your product manager skills showing. Have the customers just told us they liked the new feature, is there behavioral evidence, or do you already have five signed contracts waiting for you to build the next success? Knowing the certainty level will make you avoid poor decisions, and give you proof that customers will pay and ultimately make your company profitable.</p>
<p>Sources:</p>
<p>[1] <span style="font-style: italic;">Wilson, Laser, &amp; Stone (1982): Judging the predictors of one&#8217;s own mood: Accuracy and the use of shared theories </span></p>
<p>[2]: <span style="font-style: italic;">Weingarten, E., Chen, Q., McAdams, M., Yi, J., Hepler, J., &amp; Albarracín, D. (2016). From primed concepts to action: A meta-analysis of the behavioral effects of incidentally presented words. Psychological bulletin, 142(5), 472–497. Nisbett, Richard, &amp; Wilson, Timothy. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231-259.</span></p>
<div class="g g-21"><div class="g-single a-170"><a class="gofollow" data-track="MTcwLDIxLDEsNjA=" href="https://www.pendo.io/pendomonium/"><img decoding="async" src="https://www.mindtheproduct.com/wp-content/uploads/2024/06/MTP-Footer_ad.png" /></a></div></div>

Deciding What is Valuable: A Hierarchy to Make Your Product Profitable

Jacob de Lichtenberg

Comments

Read next