How do you decide what to prioritize or whether to believe what customers and prospects are telling you about your product? Here, I’ll explain how applying the hierarchy of value certainty, shown below, can help you to answer these questions and make your product profitable.
A core skill of product management is prioritization. For most, the end goal is to prioritize the things that will make you become the most profitable company. At the heart of your prioritization skills should lie value and effort. In B2B companies, value should ultimately generate profit (get new customers, upsell to current customers, or avoid churn), and it ends up in an equation like this.
However, not all methods are equally effective when deciding what is valuable. In fact, some methods are pretty poor. Unfortunately, it is these methods that product managers often use.
Your Skills Resume: Can you Identify Profit?
Some product managers make poor decisions because they don’t have the right skills. They identify a feature as profitable, when it isn’t. One problem is that they believe people too easily. I have a background in behavioral psychology, and one lesson I’ve learned from this field is that people don’t always know why they feel like they do[1], and they might rationalize why they say something and come up with an explanation, even though they don’t really know why[2].
Take this experiment[1], where college students completed questionnaires about their mood and other data points every day for five weeks. Researchers then asked the students what they believed had influenced their mood. Since they had the data, they could now compare the students’ subjective estimations with the verified correlations between data and mood. Look at these results:
| Predictor variable | Subjective estimate |
Objective correlation |
| Sleep | 0.33 | 0.04 |
| Friday | 0.60 | 0.05 |
| Saturday | 0.59 | 0.13 |
| Weather | 0.49 | 0.27 |
| Health | 0.51 | 0.51 |
| Relationships | 0.59 | 0.50 |
| Food | 0.16 | 0.35 |
Notice that people believe the day of the week has a massive influence. It doesn’t. And they significantly underestimate the influence of food. But they do get some things right: health and relationships.
The analogy to product management is that customers don’t always know what they want. And if you show them something that doesn’t exist, they often come up with an estimate similar to this experiment. The point is, what customers say is not enough to create certainty that something is valuable and will be profitable. The reality about why product managers fail is often related to believing what customers say.
How Something Seems Valuable, but it isn’t
When I worked at Trustpilot, we had solid evidence that consumers wanted to read reviews about products. (Currently, Trustpilot only shows company reviews). Since Trustpilot receives revenue from companies paying to collect, analyze, and promote their reviews (it’s also free with fewer features), it was important that companies found product reviews valuable to display as well. Therefore, we scheduled 20 interviews.
The feedback on the concept was stunning: 15 out of 20 companies were excited about the idea.
75% loved the idea
We asked them to confirm their interest and participate in a beta release. These are some of the confirmation emails we got:
| “Thank you for today’s meeting and presentation and I confirm our interest in both of the concepts being developed and look forward to hearing more and participating in beta trials.” | “I think it is a very good idea. The number one reason is to help us make sure that we have good customer experiences. Actually, … this adds more customer confidence in us and our products.” | “We have 7000 customers … Many of them also leave a product review on Trustpilot, which we wish would be shown publicly on Trustpilot and not only on our own website… We think this is awesome.“ |
Imagine this was your company. Doesn’t it sound like a very valuable thing to prioritize if 75% of customers want it? You might be inclined to say, let’s do it! But the product manager answer is: You don’t know!
The Hierarchy of Value Certainty
Say it
In the hierarchy I’m suggesting, if a customer tells you something is valuable, it gives you the lowest level of certainty that it will be valuable to the customer. You need more. The Trustpilot case showed 75% of customers liked our idea. But it turned out that this information was useless as soon as we did something more, something I’ll return to later in this article.
Un-shame… and Wait for Endorsement
If you interview your customers about a specific feature, rather than ask what they think about an idea or concept – lead them in the opposite direction. Most customers want an interview to be comfortable. Many customers will try to answer in a way to make the interview pleasurable. So you need to lead them away from what they think you’ll like. Here is an example:
Hi. My name is Jacob. I am going to show you a mockup of an idea the company is working on. It’s from another team, so you can be completely honest with me. I have just been tasked to research the idea. To be honest, I am not sure I fully understand it, but this is not my area…
When you show the mockups, start shaming the concept:
It just doesn’t seem like it’s their best work / best idea.
The point here is that the customer should start defending the idea. If they don’t, it’s probably not as valuable as you think.
Prioritize Over Other Valuable Features
An even better way to understand the value of an idea is to compare it to other things. If you are planning to build a Chrome extension to your product, it’s easy to ask customers to prioritize the idea over other things during interviews:
Question: Out of the following, what do you think would be most important for us to work on:
(Or you could ask the customer to rank them) |
While value/effort might be bad for some of the other suggestions, you should consider if the idea in focus is valuable enough if customers don’t rank it on top. Remember that real life is often messy, and asking for prioritization will often create messy results in contrast to the beautifully simple result of “75% of our customers love this idea” you get if you ask them. Asking customers for feedback can provide a lot of insight. But it does not provide validation.
Pretend it’s Real
Pretending that a feature exists is a great way to get validation. Advocated by people like Alberto Savoia in Pretotyping and Eric Ries in The Lean Startup, these are good methods to test if something is valuable fast, and it is a product management skill you should learn.
Let me give you an example from my days at Citrix. We looked at using Citrix’s current feature for new products, and one idea was to build an Uber-clone, an IT support site, where consumers could go to get help from multiple freelance IT support specialists. They would get instant help, so the working title was “Instant Relief”, and we even bought the domain Instant-relief.com.
As in the Trustpilot example, we set up 25 user interviews with IT freelance support specialists and consumers to get their feedback. The results were astonishing: 75% of consumers said they would use and pay for the service, and 78% of IT support specialists would sign up. But because what people say doesn’t really validate an idea, we wanted more certainty.
Instead of building anything, we simply mocked up a web page – a user interface that looked like a working page, but whoever you clicked to get support from, you connected to the same internal support specialist through a chat session, where we would pull people into Citrix’s own systems. We could get traffic to this unknown site by buying it through AdWords. This concept is called a mechanical turk.
As you may have guessed, Instant-relief.com doesn’t exist today. Cases like these demonstrate that when people believe something exists, their behavior is very different from what they say.
There are other methods for pretending something exists (like having a sales person trying to sell your idea even though it doesn’t exist). Many variations of pretending it’s real will let you measure behavior, which is the next level of the value-certainty hierarchy.
Behavior
There are many great ways of using behavioral data to know if something will be valuable.
Tracking: At Forecast, we want to change the future of project management, so that projects succeed and become profitable. While our product combines resource and project management functionality with artificial intelligence, a big part of project success relies on teamwork. We had an idea that the page for scoping out the project (meant for project managers), would actually help team members while executing the project. As we have tracking and customer data readily available at Forecast, we could quickly run a test:
Instead of asking team members if they would use this type of page, we simply selected a sample of customers and made the page accessible to them. We then tracked behavior on the page to see if the sample users were using the page after three weeks.
Tracking data only tells you about your current customers. Sometimes there might be other ways of working – but you might not be able to see this in the data.
There are other methods for looking at behaviour, Lost deals for example. If customers start engaging with your sales department and love your product, but end up not buying because you lack something, then there is behavioral proof that this something is very valuable to them. Experience it a few times, and you probably have proof that this something is something you should research.
That requires discipline from your sales department, which many startups do not have time for – though at a recent conference I attended every successful startup CEO said “if you are not disciplined in documenting the sales process, you will never succeed”.
Investment
If customers are interested enough in the new idea, they are willing to work for it. So if you can get them to invest, you are more certain that is something valuable to them.
Let’s return to the Trustpilot case. Remember that 75% of customers said they loved the idea.
We then asked the same customers to invest by sending us their product catalog. This is not a trivial task. It requires significant work from their side, and might even include others in their organization.
So how many sent us their product catalog? 3 out of 20.
15% invested
For me, this is the real number you should look at. Instead of getting overwhelmed by what people say, you should look at how many people actually invest or commit to the idea. If you are working up a business case, this is the number you should use.
Thinking about how to get people to invest is definitely a skill to keep practising.
Commitment
If there is one thing you need in order to know if something is valuable and will make your product profitable, then it is customer commitment in the form of a signed contract. That is hard to get, because few companies are willing to commit to something that doesn’t exist. But if people are really excited about something, they are also willing to commit to get it.
Before Ross Mason started Mulesoft (Valuation at IPO: $2.9 billion), the investors were able to call JP Morgan as it was ready to buy the product, if there was a company to buy from. This is an example of a potential customer being fully committed.
You should always try to get commitment! The problem is that many product managers never ask for commitment, because it can feel uncomfortable – mainly because they see it as changing the conversion from research/discovery to sales. I highly recommend moving conversations in this direction, especially when it comes to entirely new features in your product. These are the riskiest investments, and this is where you need the highest certainty.
Value: The Core of Prioritization
Part of your skillset as a product manager should be to know about prioritization frameworks: All of these have value as a core of how to prioritize. It may be known as satisfaction, impact, value, and even revenue. One of the best-known techniques is RICE prioritization from Intercom:
Using my suggested hierarchy helps you with both impact and confidence. The higher you are in the hierarchy of value certainty, the more confidence you eventually have. This is the main use. (Remember certainty about effort and reach). Evaluating the impact (which Intercom’s Sean McBride handles cleverly, but also loosely, by putting in a number between 1-3) is much easier if you are certain about the value. By using the methods higher in the hierarchy, you basically prove the value/impact.
Your Skill: Ask About the Hierarchy Level
When people say something is valuable, always ask on what level of the hierarchy there is evidence of its value. This is your product manager skills showing. Have the customers just told us they liked the new feature, is there behavioral evidence, or do you already have five signed contracts waiting for you to build the next success? Knowing the certainty level will make you avoid poor decisions, and give you proof that customers will pay and ultimately make your company profitable.
Sources:
[1] Wilson, Laser, & Stone (1982): Judging the predictors of one’s own mood: Accuracy and the use of shared theories
[2]: Weingarten, E., Chen, Q., McAdams, M., Yi, J., Hepler, J., & Albarracín, D. (2016). From primed concepts to action: A meta-analysis of the behavioral effects of incidentally presented words. Psychological bulletin, 142(5), 472–497. Nisbett, Richard, & Wilson, Timothy. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231-259.
Comments 0