post

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Successful product management in companies where it’s least understood

Measuring critical user journeys – Javier Andrés Bargas-Avila (UX Director, Google)

A new way to look at applied experiments

Product survey findings: Only 15% of users are embracing AI features 

Building high-performing product teams: The PDLC approach

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Guest Post

Overcoming the 20% feature usage challenge

Product Management

Three product leaders to learn from at #mtpcon inside Pendomonium

Karine has been in product management for about a decade, experiencing time on both the permanent and contractor sides of the fence. She has spent numerous years launching new products in start-ups and most recently has helped larger organisations like Hearst Magazines, Blinkbox (now Talk Talk TV) and the Money Advice Service with her entrepreneurial drive.

Karine Masse

<p>Don’t get me wrong, I’m an advocate of data-driven decisions and I prefer my opinions and ideas to be grounded in a robust number backed case. I can have jubilatory moments when analysing figures and seeing the positive impact of what I do, as a product manager. This is also why I continue working out the numbers myself even if there are teams of analysts who could do it for me. And this is also why I have learned to not blindly trust the numbers.</p>
<h2>Numbers are wrong more often than you think</h2>
<p>What can possibly go wrong? Well, a lot. Take a look at this article on <a href="https://www.optimizesmart.com/common-google-analytics-mistakes-that-kill-your-analysis-reporting-and-conversions/">Google Analytics common mistakes</a> or this <a href="https://inbound.org/discuss/can-we-trust-our-google-analytics-data">discussion</a> on trusting GA data. Your tracking set-up may be incorrect, the knowledge and documentation may be missing important information &#8211; for example, events may not be named in an explanatory fashion or the tracking start date not taken into account &#8211; the analyser may make errors in querying the data and then calculating results.</p>
<h2>You’re missing the full picture</h2>
<p>Numbers can be misinterpreted if the full context is not understood. For example I’ve sometimes scratched my head wondering why my conversion rate was not going up after making improvements to a purchase funnel, and then realised that the marketing team had started an acquisition campaign that meant a higher volume of our visitors were less ‘qualified’ than before, and consequently were less prone to convert on average.</p>
<p>If the reverse had happened and my conversion rate had benefited from a drop in marketing acquisition volumes, I may not have questioned it and prided myself for the full extent of the conversion rate hike.</p>
<h2>The rule of one key success metric for all devices does not work</h2>
<p>Your user behaviours may vary depending on the device used, and so should the metrics. Having worked with publishers, who do not have the ‘easy’ task of measuring against a purchase funnel, it can be challenging to find the right KPIs to worry about. And content consumption, especially, will most likely show lower KPIs on mobile, making it easy to deduce there is a problem.</p>
<p>However, sometimes the metric is just not adapted. I have once seen a same scroll depth target of 75% applied to desktop users and to mobile users. When inspecting the layout of the pages on each device, I realised that on mobile all the left hand and right hand elements of the desktop page were stacked underneath the core content, meaning that a user was only 50% down the page once they had read the full article.</p>
<h2>‘Good’ numbers may hide big UX issues</h2>
<p>It is common practice to check user experience when you realise a figure does not match expectations, for example if you identify a huge drop of conversion from page 2 to 3 in your purchase flow.</p>
<p>But what if the conversion goes up between those 2 pages because the user has missed a crucial piece of information and all they saw was to press ‘next’? Sometimes, numbers will make you think all is fine, when it is not.</p>
<p>Stats cannot replace the value of regularly watching customers use your service either through face-to-face or recorded sessions.</p>
<h2>When KPIs do not say anything useful</h2>
<p>One of the big culprits &#8211; thankfully slowly becoming irrelevant &#8211; is page views. If set as a target, someone will surely find ways to grow this KPI without any improvement in customer behaviour. And this is probably how these endless galleries of images were born, where each picture counts as a page view, or articles broken down in multiple pages, with no benefit to the users.</p>
<p>For all of these reasons, I would not take numbers for granted, and would bear the following in mind:-</p>
<ul>
<li>If possible, verify the numbers (as a QA manager would test the code) because they can create as much harm (if not more) to your business as bugs do</li>
<li>Be critical and challenge them, and not just the unfavourable ones</li>
<li>Keep adapting your metrics as you enhance your own product</li>
<li>Combine with qualitative insight</li>
</ul>
<div class="g g-21"><div class="g-single a-170"><a class="gofollow" data-track="MTcwLDIxLDEsNjA=" href="https://www.pendo.io/pendomonium/"><img decoding="async" src="https://www.mindtheproduct.com/wp-content/uploads/2024/06/MTP-Footer_ad.png" /></a></div></div>

core/freeform

How to lead data-centric product teams: Jon Mora (Chief AI Officer, Zefr)

How to be a better data-driven product manager

User insights for product decisions by Cris Valerio

Data scaling for startups by Crystal Widjaja

Make Better Product Decisions by Using Quantitative and Qualitative Data, by Brian Tran

Five Reasons Not to Trust Your Analytics Data

Karine Masse

Comments

Read next