Whether you're a data product manager or a product manager, data will be at the heart of everything you do. In this two-part series we’ll look at the importance of data and how data can help you to do a better job as a product manager.
TL;DR
- Data is both quantitative and qualitative: don’t just think about quantitative data when making data-driven decisions
- Data won’t eliminate uncertainty.
- Collect data that allows you to understand what users are doing with a product and why they are doing it.
- Data storage needs to be a balance between centralisation and usability.
- Don’t dismiss gut instinct.
Everyone in Product wants to talk about the importance of data at the moment - over the last few years Mind the Product has carried countless articles and videos about using data to make decisions and how to get the best out of your data. A few useful examples:
But first, the basics. In this post we’ll look at what being data-driven really means, what data product managers do and how their role differs from traditional product management, how you can and should collect data, and why it’s important that product managers understand how it should be stored. It’s not a list of data management or data governance frameworks or a look at the differences in data protection regulation or data security legislation around the world - more it is aimed at helping product managers to understand good practice and the discipline needed to make data-driven decisions.
What does being data driven really mean?
Let’s say at the outset that the word ‘data’ encompasses both quantitative and qualitative data - but in product management we often just think about quantitative data. This ProductTank video
Make Better Product Decisions by Using Quantitative and Qualitative Data, by Brian Tran demonstrates how important it is to consider both when making decisions.
With that in mind, we got pretty consistent answers from the senior product people we spoke to about what it means to be data driven. “You’re not relying on your intuition or gut feels to make decisions”, “you’re able to make decisions based on facts, not best guesses”, “you create certainty about decisions as they are made from hard facts, trends, and analysis” were just a few of the replies.
John Little, Vice President of Product at testing platform Centercode comments that being data-driven is about “letting the data be your map towards a destination”. Data is a powerful tool for making great decisions, if used correctly, but we should not confuse it with being data aware. John says many of the companies he deals with have a lot of data and a lot of information that they extract bits and pieces from when they make decisions. “This is not using data to drive decisions, rather it is merely being aware of data and using it in decisions. A quick test to see if you’re data driven or data aware is to trace back to the start of a product idea. If the inception is from data, there is a good chance you’re data-driven. If the initial idea cannot be attributed to specific data it’s probably not data-driven, regardless of the amount of data that’s associated with it now.”
But data won’t eliminate uncertainty
Matt LeMay is a product coach and consultant, Partner at Sudden Compass and the author of
Agile for Everybody. He adds a note of caution to the current rhetoric around data-driven product management. Much of it, he says, operates under the mistaken belief that you can predict the future and eliminate uncertainty. He says: “That's just how most people do things. I think unless we make a point of working in a different way, we all believe that the future will be a continuation of the past, that if a trend is going in a certain direction, it'll continue to go in that direction.”
At its best, data-driven product management helps us to think in systems and to recognise the interconnections and nonlinear dependencies, Matt says: “The only successful approaches I've seen to data-driven product management continue to live in uncertainty and optionality. In other words, they continue to say, here are five things, five ways this might go, we'll look at the data and see which way it's going. The ineffective data-driven product management I've seen treats a decision as a one-time thing based on what we think is going to happen, and then never reevaluates that decision.”
What is data product management, and how much of it is what product managers do?
Data product management is emerging as a distinct position as the complexity of managing data has increased and data-driven product management gains traction. A data product manager oversees how data is used in a business, treating data like a product, making sure data is used effectively.
Alison Whitehouse, Product and Organisational Agility Consultant at PA Consulting explains: “The product manager role typically uses data to make product decisions. A data product manager looks at the data strategy to support product and service decisions. They look to see that there are no blindspots for data capture which would result in not being able to capitalise on a market opportunity.”
Data management is still an essential part of any product management role and forms the basis of any product decision. John Little comments that data product management may fall solely on the product manager in a less mature business, but in mature, product-led, data-driven businesses, product teams will work with data teams to manage the data product, in a similar way that product managers work with engineering. He says: “They will define requirements (or hypothesis) and the data experts will create the data ‘product’ as defined. In a data-driven organisation, depending on where a product manager is in the product life cycle and their supporting teams, they should expect to spend around 10-25% of their time working with data.”
How can you/should you collect data?
There are lots of ways to collect data, but essentially product managers need to collect data that allows them to understand what users are doing with a product and why they are doing it. As
Justin Bauer, Chief Product Officer at Amplitude comments, you can collect all the data in the world, but it has little value if you’re not able to derive any insights from it.
Justin suggests investment in a customer data platform that creates a single centralised database of all the touchpoints that customers have with your product(s). “A common concern we hear from organisations is that their data goes from an asset to a liability when it’s not actively planned and managed,” he says. Data governance is one of the most important elements of data collection, he adds. Without it, a product team won’t be able to get an accurate view of the customer or send actionable data to downstream marketing applications. “Invest in data infrastructure that will enable you to not only ingest data, but to collect, manage, govern and analyse the
right data,” he says.
How should you store data?
As John Little says, data storage needs to be a balance between centralisation and usability. He says: “If data isn’t centralised it will take too long to bring it all together and drive decisions. However, if too much is centralised and not managed closely, it can become unwieldy and therefore unused.” He suggests a model that uses three categories with associated storage - analytics, user feedback, and development work. Then you can use features or use cases to tie the data together.
He gives an example. If you're working on updating your onboarding experience then you should identify the use cases that move someone through that experience. You should first gather the analytics that tell you what is happening at each use case. Next you pair the analytics with user feedback from beta tests and usability studies around those specific features. Finally you can create the work items necessary to make the changes based on the data.
How do you know you’re making the right decision?
Matt LeMay says he’s always asked ‘When do we know we're making the right decision?’ in his training sessions with company leaders. There isn’t an answer to this question, he says: “You can make the best decision you can at the moment, but the truth is, you won't know.”
He also cautions against dismissing gut instinct: “Your gut is likely a more complex and well-trained pattern recognition machine than whatever data system you're using. Your gut has seen the world change, and your gut can think in systems better than a lot of quantitative data analytic analytics products. I think this idea of data-driven decision-making replacing gut-driven decision-making sells them both short, because they work better in tandem.”
He summarises: “Data-driven product management means you're making decisions based on what is happening in the real world. It seems really simple. But when you're working as a product manager, you're in the world of your company. It's very easy not to live in the same world as your users or the rest of the world, and it's very easy to be very selective about the metrics you want. To me, data-driven product management is looking at that reality and all its complexity and nuance.”
Next time
In our next post we’ll look at the ins and outs of how you should present and share data with stakeholders and why it’s important to get this right, using data to help with stakeholder management, and using data to get better roadmap decisions.
Further reading
Your Data-Driven Decisions Are Probably Wrong
What is data governance and why does it matter
Why data is key in building a company that learns
