In my experience, to succeed at data-informed product management you need to master “double-think” and be comfortable with two conflicting statements: “Be data-driven” and “don’t trust your data”.
Blindly following data can lead to some bad things. Analytics products aren’t perfect – sometimes data capture gets messed up, and slight changes to code can cause all sorts of anomalies. It pays to be skeptical about your data points.
But data is important. We (D4) prefer the term “data informed“, which, for product management, we see as:
“Being data driven” + “a deep mistrust of data” = “data-informed decision making”
For us, there are three key aspects of data-informed decision making:
- Mining the raw data in front of you for patterns and trends, anomalies and fluctuations, and anything just plain weird
- Never trust what you’ve mined – data doesn’t live in a bubble so compare it to some more data and apply the context of what’s been happening with your product, customers, the weather…
- Based on the contextualized mined data, read between the lines (or data points) to validate your findings and develop a testable hypothesis to further validate
Doing just this with one of our products, we raised prices and won new customers.
Mine the raw Data (let the Data Guide you)
Our product SQLizer helps developers to convert files from various formats into SQL. It’s free up to a certain file size. If someone tries to convert a file that is too big, then they hit a paywall.
Looking through our analytics, we noticed a large number of people hitting the paywall but a relatively low conversion rate. People clearly had a need for the product because they were trying to use it. Maybe the landing page wasn’t right for the audience? Perhaps they were put off by the price? Could they just be testing some data?
Despite the supposed evidence to the contrary (never trust your data), we were fairly confident that the landing page was hitting the mark and that our prices were already on the lower end of the spectrum. So before overhauling the design and slashing prices we decided to look into some more data, this time on our current customers. It turned out we were getting a lot of new users each month for our monthly subscription but the churn rate was very high. What was going on?
Contextualize Your Data and a Testable Hypothesis (Know Your Customer’s Problem)
The problem our product solves is uniform for our audience: they all need to convert a file to SQL. But the problem differs in its regularity. Some developers work with databases all the time and have a regular need to migrate data. Other developers only need to transfer data every so often.
Based on data in our analytics we hypothesized that conversion rates were low and churn so high because not everyone needed to convert files throughout the month, they might only need to do this once every couple of months.
The data seemed to tell us that people were willing to pay the current price, even for short-term use. Maybe the inconvenience of signing up and then needing to cancel was one reason for such a low conversion rate?
We decided to test a new pricing structure:
- 24-hour pass (same price as monthly subscription fee)
- Monthly subscription (50% price increase on previous fee but grandfathering in existing users)
- Annual subscription (save 20% on month-on-month subscription)
An added benefit of a 24-hour pass is that it acts as a relatively cost-effective trial for people who may go on to sign up for a monthly subscription.
Just a couple of hours after the changes went live someone purchased a 24-hour pass. Later that day, someone else purchased a monthly subscription.
The first month of the new pricing structure was the best we’d ever had for new payments. Every month since has held this record.
Over the next three months our 24-hour pass usage continued to grow. This cannibalized monthly subscriptions somewhat (which is what we expected) but, despite overall lower numbers, month-on-month monthly subscriptions continued to grow and the churn rate fell dramatically.
Success!
Data as a Primer for Customer Emotion
Reading between the data points in our analytics helped us discover a new way people derived value from our product. While this new segment of users isn’t one that boosts overall retention, the product was attracting a group of people with this need. Sifting through our data allowed us to capitalize on this. This was a data-informed decision based on the three points of:
- Mining raw data
- Being skeptical
- Developing a testable hypothesis
While this was a data-informed decision, data can only get you so far. We needed to drill down into the human emotions behind the data. All purchases (even B2B) are made on an emotional level (thanks lizard brain!). Data can only guide you so far – you need to get creative if you want to find out (or at least hypothesize) about true intentions behind data points.
Unfortunately, analytics tools are limited in their ability to tell us exactly what people want. Even with great tech like video playback, as product managers we still need to practise the dark arts of delving into the minds of our customers.
Until some potentially-questionable analytics product is developed that can read the emotional intention of humans, you’ll need to rely on reading between the data points.
- Mine raw analytics data for patterns and trends, anomalies and fluctuations, and anything just plain weird
- But be skeptical and don’t trust what the initial data tells you. Cross-reference with more data and try to apply the context of what’s been happening with your product, customers, the weather…
- Let the data guide your thought process but be creative – just because it sounds farfetched doesn’t mean it’s not right
- Come up with a hypothesis and test that hypothesis – but be careful to only test one hypothesis per test so you can safely attribute results to the right hypothesis
If your tests are wrong, at least you now know that that particular angle is wrong and it was guided by data. You are now closer to 100% product insight that you were before. Congratulations!
Comments 1