post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Data

Measuring user experience in financial terms: A data-driven approach

How I got my job in product

How I got my job in product: Damilola Adelekan, Lead Product Manager at Remedial Health Solutions

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Why is hiring product managers so hard?

What burnout taught me: Lessons from product managers

Meet the new AI CEO: Klarna’s bold customer feedback loop bet

What I learned from building products in Africa

What does it take to build world class products today? Ezinne & Oji Udezue (Co-Authors at ProductMind, ex-CPOs of Calendly, Typeform & WP Engine)

Kevin Shanahan is Head of Growth at Peak, the self-improvement startup that offers the #1 brain training app worldwide. He oversees product development of new apps under the Peak brand, from concept to prototype to MVP to launch.

Kevin Shanahan

This is the second of two posts that deep dive on A/B tests, expanding on <a href="https://www.youtube.com/watch?v=KAFKKlFoJjU&amp;feature=youtu.be&amp;list=PLWz5rJ2EKKc_amR5JL4lSH4PHSWNUTV85">a talk I gave at Google Playtime 2016 in London</a>. In the first post I explained <a href="https://www.mindtheproduct.com/stepping-ab-tests/" target="_blank" rel="noopener noreferrer">how to step up your A/B testing</a>, and in this post I’ll look at how using a tracker spreadsheet can help you manage your A/B tests and better retain learnings from past ones. <a href="https://docs.google.com/spreadsheets/d/1wUfNShlpjLR293VwtPcMe8LbeBh-QgytOFYeAdExomY/edit#gid=179782741">Here’s a link</a> to the tracker so you can follow along and make your own copy (it’s free).
<a href="https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-1.png"><img loading="lazy" decoding="async" class="aligncenter size-full wp-image-11875" src="https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-1.png" alt="" width="1219" height="344" srcset="https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-1.png 1219w, https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-1-300x85.png 300w, https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-1-768x217.png 768w, https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-1-1024x289.png 1024w" sizes="auto, (max-width: 1219px) 100vw, 1219px" /></a>
<h3>Why you Need a Tracker</h3>
As mentioned in the first post, if you plan to run a lot of A/B tests then you should think about using a document to store test information, which I call the tracker. There are several benefits that a tracker gives you:
<ul>
<li>Forces you to think through the details of a test. Given that each test follows a similar structure, the tracker acts like a checklist to ensure you haven’t overlooked important details. This prevents costly fixes and re-runs later on.</li>
<li>Ensures everyone is on the same page. It’s easy for people to have different interpretations or recollections of what was agreed for an A/B test, but they’ll quickly identify these and converge if test information is listed in one place.</li>
<li>Gives visibility on live and upcoming tests. Testing typically affects many teams in a company, therefore each team needs to be able to see at a glance which A/B tests are running or planned so they can adjust their plans.</li>
<li>Stores details of past tests so you can easily reference them. It’s common to want to look back on old A/B tests to inform new ones, and the tracker prevents you from having to trawl through messages and project management tickets to do this.</li>
</ul>
<h3>Everyone Loves a Spreadsheet</h3>
We use a Google spreadsheet for the tracker at <a href="http://www.peak.net/">Peak</a>. Why a spreadsheet? Because spreadsheets are easy to understand, update, share, filter, synchronise, and have different layers of permissions. They may not be exciting but they’re the perfect format for the job.
<h3>The Parts of the Tracker</h3>
<a href="https://docs.google.com/spreadsheets/d/1wUfNShlpjLR293VwtPcMe8LbeBh-QgytOFYeAdExomY/edit#gid=179782741">Here’s a link</a> to the tracker that we use. It has five main parts, each of which is marked with a different coloured header.
<img loading="lazy" decoding="async" class="aligncenter size-full wp-image-11874" src="https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-2.png" alt="" width="1335" height="431" srcset="https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-2.png 1335w, https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-2-300x97.png 300w, https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-2-768x248.png 768w, https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-2-1024x331.png 1024w" sizes="auto, (max-width: 1335px) 100vw, 1335px" />
<ul>
<li>Test overview. Records what the test is, where it’s happening inside (or outside) your product, why you’re doing it, when you’re doing it, and what the status is.</li>
<li>Segment info. Describes the users in a test, including the countries they’re in, the languages they speak, and the devices they use</li>
</ul>
<img loading="lazy" decoding="async" class="aligncenter size-full wp-image-11873" src="https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-3.png" alt="" width="1335" height="418" srcset="https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-3.png 1335w, https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-3-300x94.png 300w, https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-3-768x240.png 768w, https://www.mindtheproduct.com/wp-content/uploads/2017/07/ab-testing-sheet-3-1024x321.png 1024w" sizes="auto, (max-width: 1335px) 100vw, 1335px" />
<ul>
<li>Variant info. Indicates how you will divide these users into variants and what the experience will be like for each one.</li>
<li>KPIs. Lists the metrics you will judge the success of the test on and the targets you hope to achieve if the test is successful.</li>
<li>Results and next steps. A short summary of the test results, links to more detailed analysis, and the next steps.</li>
</ul>
And that’s it! The tracker is unlikely to win an award anytime soon but we’ve found it to be a great way of staying on top of lots of tests.
If you found this post and the tracker useful, or have any questions or suggestions for improvements, I’d love to hear from you in the comments section below. Here’s the <a href="https://docs.google.com/spreadsheets/d/1wUfNShlpjLR293VwtPcMe8LbeBh-QgytOFYeAdExomY/edit#gid=179782741">link </a>to the tracker once more, which is free to copy and modify.

core/freeform

How Booking.com tackled a data quality crisis: Nimit Bhardwaj (Data Product Manager) – Product Unplugged

How to lead data-centric product teams: Jon Mora (Chief AI Officer, Zefr)

How to be a better data-driven product manager

User insights for product decisions by Cris Valerio

Data scaling for startups by Crystal Widjaja

Make Better Product Decisions by Using Quantitative and Qualitative Data, by Brian Tran

Tracking your A/B tests

Kevin Shanahan

Read more