post

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Successful product management in companies where it’s least understood

Measuring critical user journeys – Javier Andrés Bargas-Avila (UX Director, Google)

A new way to look at applied experiments

Product survey findings: Only 15% of users are embracing AI features 

Building high-performing product teams: The PDLC approach

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Guest Post

Overcoming the 20% feature usage challenge

Product Management

Three product leaders to learn from at #mtpcon inside Pendomonium

Ryan has over 14 years of experience in design, UX, research, and creative leadership. He is passionate about fostering the growth of creative individuals and scaling design teams. The challenges and experiences that come with supporting relentlessly talented people are the fuel that drives him every day.

Ryan Spanswick

<p>This article explores the value of both moderated and unmoderated user research. It outlines the differences between the two, and looks at the potential options for product managers in the light of timeline, budget, and expected revenue. And while moderated research may be expensive, it&#8217;s my view that the benefits it delivers are worth the cost. Unmoderated research can be useful in certain scenarios, but it is important to recognize its limitations.</p>
<h3>Good Usability is Paramount for any Product</h3>
<p>Every second of poor usability is an opportunity for people to abandon your product and find an alternative elsewhere. Conversely, every usability issue that is identified and fixed before your product gets in the hands of real users saves you the time and money of costly reworking or lost business.</p>
<p>If you only take one thing away from this article, it should be that:</p>
<p><strong>User research always provides value</strong></p>
<p>There’s no doubt that companies who invest in <a href="https://praxent.com/blog/5-ways-user-centered-design-can-benefit-your-bottom-line">finding and addressing usability issues</a> before they build a new product or feature will see far greater return on their product than those who don’t.</p>
<p>While user research should always be a part of the product development process, there are a couple of options when it comes to the level of investment companies make in testing and research: moderated or unmoderated user research.</p>
<h3>Over 85% of Usability Issues Identified With a Moderated Usability Study</h3>
<p>Moderated usability studies involve in-person tests with real users. The researcher recruits people who fit the user persona &#8211; and bear in mind that it can be difficult and time-consuming to recruit representative users &#8211; and schedules a time to conduct the study in real time.</p>
<p>Equipped with a script (also called a protocol) the researcher leads the user through the study, watching their interactions and body language as they work, and asking questions to get to the heart of the person’s behaviors, motivations, and experience with the product.</p>
<p>Moderated usability studies allow product teams to gather deep insights about a product or feature, including user sentiment, challenges or roadblocks users encounter, and how intuitive the experience is.</p>
<p>While the lab conditions won&#8217;t always accurately represent real life, by interacting directly with users, teams are able to more fully understand challenges, frustrations, expectations, and moments of delight experienced throughout the study.</p>
<h3>The 4 Phases of Moderated Usability Studies</h3>
<h4>1. Preparation</h4>
<p>Researchers review the interactive prototype, write scripts and protocols for recruitment and facilitation, identify key personas, and conduct dry-runs of their study to refine the process.</p>
<h4>2. Recruiting</h4>
<p>Researchers seek participants, often through an email blast with details about the research study or working with an internal recruiter or recruiting service. They follow up with respondents, schedule the tests, and confirm with participants before the study.</p>
<h4>3. Facilitation</h4>
<p>Researchers facilitate five sessions where they capture insights throughout the tests. Typically, these insights are used to refine and iterate the prototype for a follow-on study with five more users.</p>
<h4>4. Synthesis</h4>
<p>Researchers review the results of the studies, synthesizing key insights on usability issues, user sentiment, and more. They present their results and data-backed recommendations to the design team for product improvement.</p>
<h3>The Power of Moderated User Research</h3>
<p>The real power behind moderated research is the ability to dig deeper in the moment. A user’s shrug, wince, or moment of silence can uncover a truly valuable insight into issues with your product. Researchers are experts at helping users to voice what they are experiencing&#8211;saving companies from costly rework after a product or feature launches.</p>
<p>An intuitive experience can make or break a product. A well-scripted moderated usability study with just five people will typically uncover 85% to 95% of usability issues. For digital product companies of all sizes, that pays for itself in spades.</p>
<p>Take, for example, a company that intends to make $50,000 to $100,000 on each transaction with its digital product. If this company spends $10,000 upfront on moderated usability testing, it discovers 85% of user issues before development begins. The team is then able to iron out design kinks before the product is built.</p>
<p>The company saves perhaps hundreds of thousands of dollars long term by not having to go back and fix usability issues after release &#8211; not to mention the money saved by not losing customers due to a poorly designed product. After just one sale with the new product, the company has paid for its investment in moderated user research.</p>
<h3>The Cascading Effect of Moderated Usability Studies</h3>
<p>According to a <a href="https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/" target="_blank" rel="noopener noreferrer">statistical analysis by user experience experts at Nielson Norman</a>, a user test with only five people will identify 85% of usability issues. After iterating the prototype from insights uncovered in the first study, conducting a second study has a cascading benefit.</p>
<p>Anything that was left unchanged in the iteration is now tested a second time, bringing the total identified usability issues up to 95% for the unchanged portions of the experience. Paired with the 85% of usability issues uncovered in the changed areas, this cascade increases the overall confidence in the usability of the product. After 10 sessions, it becomes unnecessary to test the product with any more users. Additional tests will simply edge closer to 100% at a much lower rate of return.</p>
<p>Researchers can maximize the return per user tested and achieve the 95% by tapping into the cascading effect:</p>
<ol>
<li>Test the product on five users to identify 85% of usability issues.</li>
<li>Refine the design, addressing every found problem.</li>
<li>Test again with five new people to identify new problems from changed features while narrowing the gap on unchanged features by an additional 10%.</li>
</ol>
<p>With this three-step process, you can optimize the impact of each round of tests, accessing highly nuanced results while discovering as many issues as possible.</p>
<h3>Unmoderated User Research is Cheaper</h3>
<p>Unmoderated user research removes the in-person contact between user and researcher. It’s often an automated process that provides users with tasks to work through, generating automatically recorded results or recording the session for later analysis.</p>
<p>Unmoderated user studies can require less time and effort than moderated user studies, <a href="https://www.mindtheproduct.com/quantifying-qualitative-research-by-leisa-reichelt/">and this makes it a popular approach</a>. Users can complete tests from anywhere, in their own time, but the researcher is not personally involved in facilitating the tests to ask questions that dig deeper.</p>
<h3>4 Examples of Unmoderated User Studies</h3>
<p>With these four types of automated, unmoderated user studies, product teams can collect data on several key areas of interest like information architecture, content efficacy, bounce rate, brand communication, design approach, and user motivations.</p>
<h4>1. First-Click Test</h4>
<p>Measure the effectiveness of information hierarchy, links, content placement, visual layout, and content hierarchy by tracking how users complete tasks with a single click.</p>
<h4>2. 5-Second Test</h4>
<p>Optimize content, stress test your design approach, and reduce bounce rates on your site by measuring people’s first impressions of a single screen.</p>
<h4>3. Preference Test</h4>
<p>Refine content, visual design, branding, and more by measuring user affinity and sentiment to multiple options.</p>
<h4>4. Design Surveys</h4>
<p>End internal debates, gain deeper insights and avoid costly missteps by asking users questions about your assumptions.</p>
<h3>Quick Insights With Unmoderated User Research</h3>
<p>Unmoderated user studies allow product teams to gather insight quickly. Having users work through quick sessions on their own provides access to actionable data with a quicker turnaround than moderated research and often reduces the investment required to conduct the study.</p>
<p>The difficulty with unmoderated user research is that it doesn’t provide the opportunity for researchers to closely observe and question people as they test. Researchers are forced to make some assumptions based on what they can infer from their observations.</p>
<p>Here is an example scenario of an unmoderated user study:</p>
<ol>
<li>Researchers seek a large group of users &#8211; anywhere from 100 to 1,000 or more people.</li>
<li>They equip the users with an unmoderated first-click test to be completed remotely.</li>
<li>The automated test presents users with a task to complete with a single click.</li>
<li>The test automatically sends the researchers a heat map with data about what users clicked when presented with a task.</li>
</ol>
<p>While this test presents researchers with data about where people clicked first, it fails to provide deeper insight into why they clicked it. Researchers must either make assumptions or conduct additional studies to understand the findings.</p>
<p>Product teams who rely on unmoderated user studies &#8211; whether they are first-click tests, preference tests, or another type of unmoderated test &#8211; will struggle to gain real insight about the reasons behind user choices and behaviors and will spend more time guessing at solutions as a result.</p>
<h3>Top 3 Takeaways on Moderated vs Unmoderated User Research</h3>
<p>Generally, moderated user research provides better long-term results and can save money and time in the long run. Unmoderated user research may cost less in time and money upfront, but at the expense of long-term savings and product quality.</p>
<h4>Takeaway 1: Don’t Skimp on User Research</h4>
<p>Companies building digital products should not hesitate to invest in user research. There is no question that the research brings immense value to the product at launch and beyond.</p>
<h4>Takeaway 2: Some Research is Better Than Nothing</h4>
<p>While there is a higher value in moderated than unmoderated research, either is better than nothing. Choose wisely depending on what kind of data you are trying to attain. There are countless examples in the market of product companies that spent hundreds of thousands of dollars to correct usability issues that could have been easily identified before development. All user research provides a return.</p>
<h4>Takeaway 3: It’s Easier to do Research than it is to Rework</h4>
<p>In most cases, research is left out of the process to reduce budget, increase speed to deliver, or both. Companies who do this typically invest significantly more time and money after launch fixing issues and scrambling to hang on to customers.</p>
<div class="g g-21"><div class="g-single a-170"><a class="gofollow" data-track="MTcwLDIxLDEsNjA=" href="https://www.pendo.io/pendomonium/"><img decoding="async" src="https://www.mindtheproduct.com/wp-content/uploads/2024/06/MTP-Footer_ad.png" /></a></div></div>

core/freeform

Product Management Skills

How to keep your head about generative AI (when everyone is losing theirs) by Claire Woodcock

Why Moderated User Research is Worth the Cost

Ryan Spanswick

Comments

Read next