post

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Successful product management in companies where it’s least understood

Measuring critical user journeys – Javier Andrés Bargas-Avila (UX Director, Google)

A new way to look at applied experiments

Product survey findings: Only 15% of users are embracing AI features 

Building high-performing product teams: The PDLC approach

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Guest Post

Overcoming the 20% feature usage challenge

Product Management

Three product leaders to learn from at #mtpcon inside Pendomonium

Data-driven Product Manager with an entrepreneurial background and a bias for action. Team player with a passion for collaborative design, ideation, and good user experience. Consistent track record of driving user and revenue growth through experimentation.

Bartosz Mozyrko

<h2><img loading="lazy" decoding="async" class="aligncenter wp-image-18924 size-full" src="//wp-content/uploads/2020/02/Using-smoke-tests-to-validate-your-ideas.png" alt="Using smoke tests to validate your ideas" width="1508" height="576" srcset="/wp-content/uploads/2020/02/Using-smoke-tests-to-validate-your-ideas.png 1508w, /wp-content/uploads/2020/02/Using-smoke-tests-to-validate-your-ideas-300x115.png 300w, /wp-content/uploads/2020/02/Using-smoke-tests-to-validate-your-ideas-768x293.png 768w, /wp-content/uploads/2020/02/Using-smoke-tests-to-validate-your-ideas-1024x391.png 1024w" sizes="(max-width: 1508px) 100vw, 1508px" /></h2>
It&#8217;s said that <a href="https://www.information-age.com/projects-continue-fail-alarming-rate-123470803/" target="”_blank”" rel="”noopener” noopener noreferrer">more than two-thirds of software projects fail to deliver expected results.</a> As a result, the team at <a href="https://uniregistry.com/" target="”_blank”" rel="”noopener” noopener noreferrer">Uniregistry </a>decided to take a new approach to product validation in an attempt to avoid the same fate. Here&#8217;s how they used smoke tests to explore a new opportunity.
<h2>Overview</h2>
This case study describes a part of a larger product discovery project that I led as a product manager at <a href="https://uniregistry.com/">Uniregistry</a>, a platform that enables easy buying and management of domains and complementary products (online presence, business apps). The company&#8217;s products are used by 30,000 customers with over a million domains under our management. The company started as a domain registrar targeting independent entrepreneurs around the world and domain investors who demanded advanced tools to manage hundreds (and sometimes hundreds of thousands) of domains. Recently we’ve seen a new trend &#8211; that more of our customers use just a single domain and are freelancers, small or micro businesses. We decided to explore this new opportunity to grow our business beyond its core offering and to refresh our understanding of all customer segments. This case study focuses on Lean Product Validation, but the chart below gives an overview of our entire customer discovery process and the steps we followed.
<figure style="width: 885px" class="wp-caption aligncenter"><a href="//wp-content/uploads/2020/02/0d0d3eaf-3beb-48e3-96d3-855b23c05ff4_pasted20image200.png" target="_blank" rel="noopener noreferrer"><img loading="lazy" decoding="async" src="//wp-content/uploads/2020/02/0d0d3eaf-3beb-48e3-96d3-855b23c05ff4_pasted20image200.png" alt="The customer discovery process at Uniregistry" width="885" height="441" /></a><figcaption class="wp-caption-text">Core team: Product Manager, Product Designer, and a Full-stack Developer (click to enlarge)</figcaption></figure>
<h2>The Approach</h2>
It&#8217;s because <a href="https://www.information-age.com/projects-continue-fail-alarming-rate-123470803/">so many software projects fail to deliver expected results</a>, that we d<img loading="lazy" decoding="async" class="aligncenter size-unit-news-featured-large wp-image-18911" src="//wp-content/uploads/2020/02/c59dc694-7fa7-4f56-9526-63491eff2c5b_pasted20image200-570x320.png" alt="Uniregistry trying to cross-sell other products" width="570" height="320" srcset="/wp-content/uploads/2020/02/c59dc694-7fa7-4f56-9526-63491eff2c5b_pasted20image200-570x320.png 570w, /wp-content/uploads/2020/02/c59dc694-7fa7-4f56-9526-63491eff2c5b_pasted20image200-285x160.png 285w" sizes="(max-width: 570px) 100vw, 570px" />ecided to go with <a href="https://en.wikipedia.org/wiki/Smoke_testing_(lean_startup)">smoke tests</a>. These serve not only a useful research tool to enable us to learn more about our new customer segment, but also provide us with the knowledge about which product could be most successful commercially.
We chose three product ideas:
<ul>
<li>One Page Website</li>
<li>Advanced Website Builder</li>
<li>Web Performance Analytics</li>
</ul>
Each of these ideas was at a different stage of development. One Page Website functioned as a free feature of the Uniregistry mobile app and was deeply hidden in it. Advanced Website Builder was a white-label third-party solution that we could potentially integrate with. Web Performance Analytics was an early startup idea from the pitch deck of a partner company specializing in SEO.
We decided to launch landing pages for every product focused around a strong call-to-action, “Get Started Today”, then to bring quality traffic to the pages and monitor how they perform. Behind the scenes, none of the products existed.
It was important for us to create the impression that all the products were available to purchase.
We wanted to compare conversion funnels against the three products (between a product page and the first step of a shopping cart, and between the first and last steps of the cart). Equipped with such knowledge we could conclude which products would have greater sales potential, perhaps even greater than the products we were already offering. We would then be able to estimate the expected revenue for each product (monthly, yearly).
We ran the tests on paying customers. The biggest challenge internally was to educate customer-facing teams that our brand reputation will not be damaged if the products weren’t available to purchase. The project was also given a codename, Project Tinder, and a visual identity, shown below.
<a href="//wp-content/uploads/2020/02/2678daa7-f81b-4c48-a0e5-0d217b9e2468_pasted20image200.png" target="_blank" rel="noopener noreferrer"><img loading="lazy" decoding="async" class="aligncenter" src="//wp-content/uploads/2020/02/2678daa7-f81b-4c48-a0e5-0d217b9e2468_pasted20image200.png" alt="Uniregistry gave it's project the nickname Uni" width="420" height="420" /></a>
The codename refers to the mobile dating app that allows you to make new connections, as the goal of the project was to get to know our customers better. The codename was intended to arouse interest, to encourage conversation, and increase the chance that the learnings from the project would spread quickly through the company.
Every month, approximately 1,500 customers make at least one paid transaction on our site.
We wanted to redirect some of that high-quality traffic to our dummy pages. The last domain purchase step (Purchase Complete) was modified for the duration of the project to show three new call-to-action elements displayed in random order to each user.
<figure style="width: 885px" class="wp-caption aligncenter"><a href="//wp-content/uploads/2020/02/c59dc694-7fa7-4f56-9526-63491eff2c5b_pasted20image200.png" target="_blank" rel="noopener noreferrer"><img loading="lazy" decoding="async" src="//wp-content/uploads/2020/02/c59dc694-7fa7-4f56-9526-63491eff2c5b_pasted20image200.png" alt="" width="885" height="737" /></a><figcaption class="wp-caption-text">Step 1. Making an impression &#8211; we try to cross-sell other products (click to enlarge)</figcaption></figure>
Each product page provided clear descriptions of value-adds but also enabled an easy way to explore two other products being tested (“Other Products You May Like”).
<figure style="width: 584px" class="wp-caption aligncenter"><a href="//wp-content/uploads/2020/02/9a7e5450-170f-4d7b-928f-a208f8b767ef_pasted20image200.jpg" target="_blank" rel="noopener noreferrer"><img loading="lazy" decoding="async" src="//wp-content/uploads/2020/02/9a7e5450-170f-4d7b-928f-a208f8b767ef_pasted20image200.jpg" alt="" width="584" height="1600" /></a><figcaption class="wp-caption-text">Step 2. example: Advanced Website Builder (click to enlarge)</figcaption></figure>
From the product page, customers were redirected to a modified version of our shopping cart. We monitored how many people progressed further, and also tested conversion on yearly plans with a discount.
<figure style="width: 885px" class="wp-caption aligncenter"><a href="//wp-content/uploads/2020/02/0232e5c4-4949-4d25-9025-a808cf7b371e_pasted20image200.png" target="_blank" rel="noopener noreferrer"><img loading="lazy" decoding="async" src="//wp-content/uploads/2020/02/0232e5c4-4949-4d25-9025-a808cf7b371e_pasted20image200.png" alt="" width="885" height="305" /></a><figcaption class="wp-caption-text">Step 3. Modified shopping cart (single-product) (click to enlarge)</figcaption></figure>
For the next step, we provided a fairly general explanation of why the product was unavailable, and we also asked users to leave their contact details. Reaching this screen was considered a high-quality conversion and flagged as likely sale.
<figure style="width: 885px" class="wp-caption aligncenter"><a href="//wp-content/uploads/2020/02/b84587f7-ed75-49c5-8d86-a0db9b915915_pasted20image200.png" target="_blank" rel="noopener noreferrer"><img loading="lazy" decoding="async" src="//wp-content/uploads/2020/02/b84587f7-ed75-49c5-8d86-a0db9b915915_pasted20image200.png" alt="" width="885" height="626" /></a><figcaption class="wp-caption-text">Step 4. Sorry, the product is temporarily unavailable (click to enlarge)</figcaption></figure>
Our customer success team feared a flood of questions and complaints from dissatisfied customers due to this screen. Fortunately, we didn’t get any complaints during the project.
Then, we displayed a price sensitivity survey that allowed us to understand which aspects of the product&#8217;s value proposition were relevant to the customer. We knew that at this stage very few people would leave us additional information, but even a few answers would be enough to detect trends in preferences. We used <a href="https://developer.surveymonkey.com/" target="”_blank”" rel="”noopener” noopener noreferrer">an integration with SurveyMonkey</a> to implement it quickly and cost-effectively.
<figure style="width: 885px" class="wp-caption aligncenter"><a href="//wp-content/uploads/2020/02/a454c345-79ab-4c17-bb59-688ae58be2b0_pasted20image200.png" target="_blank" rel="noopener noreferrer"><img loading="lazy" decoding="async" src="//wp-content/uploads/2020/02/a454c345-79ab-4c17-bb59-688ae58be2b0_pasted20image200.png" alt="" width="885" height="806" /></a><figcaption class="wp-caption-text">Step 5. Tradeoff feature analysis survey embedded into the checkout (click to enlarge)</figcaption></figure>
At the last step, we thanked the user and displayed information to close the browser window.
<figure style="width: 885px" class="wp-caption aligncenter"><a href="//wp-content/uploads/2020/02/29a34a22-49f9-41b8-bbc6-5d1a3b70c15f_pasted20image200.png" target="_blank" rel="noopener noreferrer"><img loading="lazy" decoding="async" src="//wp-content/uploads/2020/02/29a34a22-49f9-41b8-bbc6-5d1a3b70c15f_pasted20image200.png" alt="" width="885" height="555" /></a><figcaption class="wp-caption-text">Step 6. The last step of the user journey (click to enlarge)</figcaption></figure>
<h2>The Results</h2>
We used <a href="https://www.woopra.com/" target="”_blank”" rel="”noopener” noopener noreferrer">Woopra</a> as the main analytical tool in our tests. We could see how far in the conversion funnel each segment went, where users stopped, how much time was spent on individual interactions and when users decided to abandon the process.
<figure style="width: 885px" class="wp-caption aligncenter"><a href="//wp-content/uploads/2020/02/fec1e583-58f6-48f3-9cd6-eb6f1352579a_pasted20image200.png" target="_blank" rel="noopener noreferrer"><img loading="lazy" decoding="async" src="//wp-content/uploads/2020/02/fec1e583-58f6-48f3-9cd6-eb6f1352579a_pasted20image200.png" alt="" width="885" height="504" /></a><figcaption class="wp-caption-text">Woopra: an example conversion funnel (click to enlarge)</figcaption></figure>
Two products generated considerable interest:
<ul>
<li>47.36% &#8211; One Page Website (OnePage Lander)</li>
<li>42.98% &#8211; Advanced Website Builder (WBuilder Lander)</li>
</ul>
Web Performance Analytics (WDoctor Lander) did not seem appealing to our customers.
Half of the users who chose the Advanced Website Builder on step 2 (product page) decided to explore the offering further and ended up on the One Page Website product page and a significant number of these users decided to purchase it. It was a valuable product insight.
Interestingly, some of the users committed to purchasing Advanced Website Builder with annual plans for hundreds of dollars before even seeing the product. This was a nice surprise for the team (the pages were convincing) but also helped us estimate better potential revenue for such a product line.
<figure style="width: 885px" class="wp-caption aligncenter"><a href="//wp-content/uploads/2020/02/366ae859-def9-46bf-ab29-f1d28cb5017d_pasted20image200.png" target="_blank" rel="noopener noreferrer"><img loading="lazy" decoding="async" src="//wp-content/uploads/2020/02/366ae859-def9-46bf-ab29-f1d28cb5017d_pasted20image200.png" alt="" width="885" height="518" /></a><figcaption class="wp-caption-text">Step 4. High-quality conversion on a product (click to enlarge)</figcaption></figure>
85% of users interested in buying any product (Step 3) were interested in the One Page Website (OnePage Lander).
Using the <a href="https://www.priceintelligently.com/blog/bid/193618/You-re-Leaving-Out-the-Customer-How-to-Quantify-Your-Value-Propositions" target="”_blank”" rel="”noopener” noopener noreferrer">Tradeoff Feature Analysis</a> approach, helped to clarify which features were most valuable to our users and why.
<figure style="width: 885px" class="wp-caption aligncenter"><a href="//wp-content/uploads/2020/02/527cca8a-bef4-49aa-9a47-ba87e676d54b_pasted20image200.png" target="_blank" rel="noopener noreferrer"><img loading="lazy" decoding="async" src="//wp-content/uploads/2020/02/527cca8a-bef4-49aa-9a47-ba87e676d54b_pasted20image200.png" alt="" width="885" height="408" /></a><figcaption class="wp-caption-text">One Page Website: tradeoff feature analysis survey results (click to enlarge)</figcaption></figure>
The team and stakeholders were happy with all the collected data and product insights.
After customer discovery, we conducted internal workshops with the COO, head of customer success, head of design, product marketing, account management and created a matrix of strengths and weaknesses for various scenarios.
We considered focusing on two groups of customers: companies employing between one and five people or companies employing more than five people. This difference, although seemingly small, had significant implications for our company, both in terms of product development, strategic positioning, and marketing activities.
We took into account criteria such as market size, entry barriers, profitability, competitive environment, customer acquisition costs, product stickiness, and various internal features of our organization. We needed to agree with the direction we were going in and to make sure that everyone understood the arguments behind the decision that was eventually made (which is not synonymous with &#8220;agreeing&#8221; with them).
<h2>Conclusion</h2>
The knowledge from the discovery process helped to make a decision in allocating company resources to advance the One Page Website to an independent product. The product has been released and sales are much as we had estimated during the discovery phase.
Although the results from our discovery project were encouraging, the launch of the new product won’t guarantee its success. The validation process was just the beginning, as the product now transitions from an introductory stage to the growth stage. The goal of our team will be to enable more entrepreneurs an easier way to start and manage their online presence to grow their business.
However, by going through the whole product discovery process we are in a much better position. After rolling-out the One Page Website, the next steps are to back it up with more research to uncover how the solution could solve the customer pains better and optimize for stronger product stickiness (customer retention) together with rolling out more advanced value-adds (pricing plan optimization).
I hope that the described case study will help you carry out effective tests, facilitate decision making and increase your product&#8217;s chance of success &#8211; good luck!
<a href="https://www.mindtheproduct.com/write-for-us/case-studies/" target="_blank" rel="noopener noreferrer"><img loading="lazy" decoding="async" class="aligncenter wp-image-18926 size-full" src="//wp-content/uploads/2020/02/Submit-a-case-study-to-mind-the-product.png" alt="" width="1514" height="392" srcset="/wp-content/uploads/2020/02/Submit-a-case-study-to-mind-the-product.png 1514w, /wp-content/uploads/2020/02/Submit-a-case-study-to-mind-the-product-300x78.png 300w, /wp-content/uploads/2020/02/Submit-a-case-study-to-mind-the-product-768x199.png 768w, /wp-content/uploads/2020/02/Submit-a-case-study-to-mind-the-product-1024x265.png 1024w" sizes="(max-width: 1514px) 100vw, 1514px" /></a>
<div class="g g-21"><div class="g-single a-170"><a class="gofollow" data-track="MTcwLDIxLDEsNjA=" href="https://www.pendo.io/pendomonium/"><img decoding="async" src="https://www.mindtheproduct.com/wp-content/uploads/2024/06/MTP-Footer_ad.png" /></a></div></div>

How Uniregistry Used Smoke Testing in Product Validation: A Case Study

Bartosz Mozyrko

Comments

Read next