post

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Successful product management in companies where it’s least understood

Measuring critical user journeys – Javier Andrés Bargas-Avila (UX Director, Google)

A new way to look at applied experiments

Product survey findings: Only 15% of users are embracing AI features 

Building high-performing product teams: The PDLC approach

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Guest Post

Overcoming the 20% feature usage challenge

Product Management

Three product leaders to learn from at #mtpcon inside Pendomonium

As Director of Product Management at Optimizely, Jon Noronha is responsible for leading a team of product managers with a goal to discover new ways for companies to experiment more across websites, apps and every level of the stack. Prior to joining Optimizely, Jon coordinated engineering teams across Seattle and Beijing to rethink visual search at Microsoft. As part of Bing’s Image Search team at Microsoft, Jon developed cutting-edge technology in machine learning, distributed systems, and image processing and combined it with great design based on usability studies, constant A/B testing, and quantitative analysis. Jon has a bachelor’s degree in Computer Science from Harvard University.

Jon Noronha

<p>As <a href="https://www.mindtheproduct.com/what-exactly-is-a-product-manager/" target="_blank" rel="noopener noreferrer">product managers</a>, our job is to determine what matters most to our customers. At any given time, there could be 100 really great ideas that need implementing, but realistically we can only complete two or three per month.</p>
<p>We have to ask ourselves: which three do we build? How deep do we go? And how much time should we spend iterating on a product that launched last month versus moving onto something new? There’s a high <a href="https://medium.com/@hellomeets/the-economics-behind-building-a-good-product-c1345de017f6" target="_blank" rel="noopener noreferrer">opportunity cost</a> because whichever three ideas we choose to initiate come at the expense of the other 97.</p>
<p>In this post, I’ll discuss the foundations of effective product experimentation, and how you can implement it into your business strategy today.</p>
<h3>Product Experimentation is Changing</h3>
<p>In a world where <a href="https://www.chiefmarketer.com/customer-experiences-need-exceed-expectations-gartner/" target="_blank" rel="noopener noreferrer">customers expect</a> more from their interactions with digital companies than ever before, product teams are turning to experimentation to answer these questions accurately. Experimenting lets us measure the impact of the products we’re building quantitatively, by testing them before and after release.</p>
<figure id="attachment_21671" aria-describedby="caption-attachment-21671" style="width: 1024px" class="wp-caption aligncenter"><a href="//wp-content/uploads/2018/10/AB-testing-image-shutterstock.jpg"><img loading="lazy" decoding="async" class="wp-image-21671 size-large" src="//wp-content/uploads/2018/10/AB-testing-image-shutterstock-1024x683.jpg" alt="A/B testing" width="1024" height="683" srcset="/wp-content/uploads/2018/10/AB-testing-image-shutterstock-1024x683.jpg 1024w, /wp-content/uploads/2018/10/AB-testing-image-shutterstock-300x200.jpg 300w, /wp-content/uploads/2018/10/AB-testing-image-shutterstock-768x512.jpg 768w" sizes="(max-width: 1024px) 100vw, 1024px" /></a><figcaption id="caption-attachment-21671" class="wp-caption-text">Jon believes we&#8217;re now seeing a shift from the idea of A/B testing (Image: Shutterstock)</figcaption></figure>
<p>We’re now seeing a shift from the idea of A/B testing as a narrow technique focused on conversion rates and landing pages towards experimentation as a broader mindset of how an entire business runs. By evaluating ideas and making decisions based on data, product managers and their engineers can eliminate the guesswork and avoid wasted time and effort.</p>
<h3>Why Agile is an Essential Prerequisite to Experimentation</h3>
<p>When experimenting, nothing is set in stone. To implement product experimentation effectively, your team needs to constantly refine interfaces, add features and test performance so you can adapt to changing customer requirements in real-time — luckily, <a href="https://explore.versionone.com/state-of-agile/versionone-11th-annual-state-of-agile-report-2" target="_blank" rel="noopener noreferrer">94% of you are</a> already practising agile today. Experimentation is the key to agility because it makes your team fearless and gives them the confidence they need to <a href="https://www.oreilly.com/ideas/the-key-to-agile-data-science-experimentation" target="_blank" rel="noopener noreferrer">quickly pivot</a> when things aren’t working out.</p>
<h3>Make Experimentation Part of Your Product DNA</h3>
<p>Today, the most innovative companies, like <a href="https://www.fastcompany.com/3063846/why-these-tech-companies-keep-running-thousands-of-failed" target="_blank" rel="noopener noreferrer">Amazon, Netflix and Facebook, run thousands of experiments annually</a>. They have embedded experimentation into their culture and use data-driven decisions to implement and create new products. As a team leader, the best thing you can do is to encourage your teams to feel comfortable with failure. Empower all your employees to think and act creatively, and make sure they understand that experimentation is a worthy investment. While you’ll want to have the right infrastructure in place to run experiments and have engineering teams focused on improving that infrastructure, it’s important to remember that experiments can be run by <em>anyone</em>. To make this work, every employee needs to feel like they can get involved and that it’s encouraged to spend time working on projects they are passionate about.</p>
<h3>Set Realistic Experimentation Goals</h3>
<p>While running thousands of experiments may not be realistic for the size of your team, depending on how many developers you have, determine what it would take to run <em>one experiment per developer per quarter</em>. As you’re building out and defining a Product Requirements Document (PRD), you should always be asking yourselves what experiments you’ve <em>already run </em>that justify building this new product, and what experiments you will run to validate the idea after it launches. By setting realistic experimentation goals, your team will be encouraged to take risks, be curious, and experiment as a day-to-day process. As results from the experiments are reported, teams can use the data to make decisions and understand key learnings and adjust course if needed.</p>
<figure id="attachment_21672" aria-describedby="caption-attachment-21672" style="width: 1024px" class="wp-caption aligncenter"><a href="//wp-content/uploads/2018/10/Discussing-experiementation-goals-image-shutterstock.jpg"><img loading="lazy" decoding="async" class="wp-image-21672 size-large" src="//wp-content/uploads/2018/10/Discussing-experiementation-goals-image-shutterstock-1024x683.jpg" alt="Meeting to Discuss experimentation goals" width="1024" height="683" srcset="/wp-content/uploads/2018/10/Discussing-experiementation-goals-image-shutterstock-1024x683.jpg 1024w, /wp-content/uploads/2018/10/Discussing-experiementation-goals-image-shutterstock-300x200.jpg 300w, /wp-content/uploads/2018/10/Discussing-experiementation-goals-image-shutterstock-768x512.jpg 768w" sizes="(max-width: 1024px) 100vw, 1024px" /></a><figcaption id="caption-attachment-21672" class="wp-caption-text">Determine what level of experimentation is realistic for your team (Image: Shutterstock)</figcaption></figure>
<p>The bottom line is, if you’re going to have an experimentation-first culture, you have to look at every failure as a learning opportunity. Create a hypothesis and test it. If it’s not valid, move on quickly to the next one and learn from it.</p>
<h3>Establish Processes and Technology to Make Experimentation Easy</h3>
<p>As part of the culture of experimentation, you also need to implement some key processes. Here are some different ways to approach this.</p>
<h4>Mitigate Risk With Gradual Rollouts</h4>
<p>Let’s say you spend nine months building a product. Instead of automatically rolling it out to 100% of your user base and hoping for the best, today we’re seeing more and more teams implement a gradual rollout with the opportunity to roll it back to safety if something doesn’t land as planned. With a gradual rollout, you can slowly introduce a new feature to a small percentage of customers as a way to monitor key engineering metrics like performance and bugs.</p>
<p>For example,<a href="https://www.businessinsider.com/netflix-testing-ads-for-original-content-2018-8" target="_blank" rel="noopener noreferrer"> Netflix recently began experimenting</a> with airing ads between episodes. Its users expressed concerns about being served commercials, so Netflix decided to gradually test whether surfacing recommendations between episodes will be positively received. By taking this strategy, Netflix is mitigating any risk in making sure this new feature won’t degrade the user experience. On the flip side, <a href="http://www.netimperative.com/2018/08/vue-cinema-hit-by-backlash-for-online-payments-delay/" target="_blank" rel="noopener noreferrer">Vue Cinema recently received a massive amount of public backlash</a> when customers were entered into a queue on its website to complete transactions, leading them to believe that the website had crashed. If Vue had experimented with this new feature by using a gradual rollout, it would have seen how customers really viewed the experience, and it could have identified what wasn’t working and adjusted its approach.</p>
<h4>Try the Painted Door Test</h4>
<p>The <a href="https://medium.com/@crstanier/a-product-managers-guide-to-painted-door-tests-a1a5de33b473" target="_blank" rel="noopener noreferrer">painted door test</a> is a great, low-investment way to gather data from your customers and validate the features you decide to build. Instead of building a feature you aren’t sure your customers will care about, <a href="https://help.optimizely.com/Ideate_and_Hypothesize/Integrating_Full_Stack_into_the_product_development_lifecycle#Discover_what_your_customers_want" target="_blank" rel="noopener noreferrer">build the suggestion of a feature</a> or solution and measure how many people try to use it. When a customer clicks on the dummy, you can auto-populate a message to say something like “<em>Thanks, we’re thinking of building this feature. Could a product manager get in touch to learn more about your needs?</em>” From there, your team can learn more about what they’re specifically looking for, which can then dramatically change the course of the product direction.</p>
<figure id="attachment_21670" aria-describedby="caption-attachment-21670" style="width: 1024px" class="wp-caption aligncenter"><a href="//wp-content/uploads/2018/10/painted-door-test-image-shutterstock.jpg"><img loading="lazy" decoding="async" class="wp-image-21670 size-large" src="//wp-content/uploads/2018/10/painted-door-test-image-shutterstock-1024x576.jpg" alt="two painted doors" width="1024" height="576" srcset="/wp-content/uploads/2018/10/painted-door-test-image-shutterstock-1024x576.jpg 1024w, /wp-content/uploads/2018/10/painted-door-test-image-shutterstock-300x169.jpg 300w, /wp-content/uploads/2018/10/painted-door-test-image-shutterstock-768x432.jpg 768w, /wp-content/uploads/2018/10/painted-door-test-image-shutterstock-285x160.jpg 285w, /wp-content/uploads/2018/10/painted-door-test-image-shutterstock-570x320.jpg 570w" sizes="(max-width: 1024px) 100vw, 1024px" /></a><figcaption id="caption-attachment-21670" class="wp-caption-text">Using the Painted Door Test, give your customers options to choose from (Image: Shutterstock)</figcaption></figure>
<p>For example, we recently had two different features we were considering at Optimizely, so we ran painted doors for both of them over the same time period and to the same number of people. The results showed that one of them got about 65 people to click on it and the other only got three. While it’s not the be-all and end-all, this gave us an indicator that we might radically get different value out of these two products.</p>
<p>It’s not uncommon for teams to spend months of time and millions of dollars of engineering effort, all to build an app or a feature that nobody uses. Painted doors are an easy way to avoid this risk. They get you direct feedback from your customers, immediate results, and qualitative insights you can use to re-prioritize engineering resources.</p>
<h3>Conclusion</h3>
<p>The best products and the most successful companies may seem like they were predestined, but anything worth talking about requires iteration. Product teams that recognize the power of experimentation will be well-positioned for the future. By encouraging your teams to set simple goals, share new ideas and test them at a quick pace, you’ll be able to use experimentation as a way to improve all facets of your businesses.</p>
<h3>Further Reading</h3>
<ul>
<li><a href="https://www.mindtheproduct.com/product-experimentation-pitfalls-and-how-to-avoid-them-by-jon-noronha/" target="_blank" rel="noopener noreferrer">Product Experimentation Pitfalls and How to Avoid Them by Jon Noronha</a></li>
<li><a href="https://www.mindtheproduct.com/get-team-experimenting-using-one-little-word/" target="_blank" rel="noopener noreferrer">Get Your Team Experimenting More by Using One Little Word…</a></li>
<li><a href="https://www.mindtheproduct.com/experimenting-with-lean-tristan-kromer-on-better-product-process/" target="_blank" rel="noopener noreferrer">Lean experiments: Tristan Kromer on better product process</a></li>
<li><a href="https://www.mindtheproduct.com/experiments-101/" target="_blank" rel="noopener noreferrer">Everything a product manager needs to know about experiments</a></li>
</ul>
<div class="g g-21"><div class="g-single a-170"><a class="gofollow" data-track="MTcwLDIxLDEsNjA=" href="https://www.pendo.io/pendomonium/"><img decoding="async" src="https://www.mindtheproduct.com/wp-content/uploads/2024/06/MTP-Footer_ad.png" /></a></div></div>

How do you Implement Effective Product Experimentation?

Jon Noronha

Comments

Read next