Get Comfortable Breaking Your Product by Rik Higham

BY James Gadsby Peet ON DECEMBER 21, 2018

Six years ago at Skyscanner there was no experimentation: now, there might be 500 tests running at any one time. In this #mtpcon London talk, Skyscanner’s Principal Product Manager Rik Higham looks at how experimentation can be a strategic capability for an organisation.

First you should change your approach to failure – rather than look at failure as from something that didn’t work, consider that you’ve only failed if you failed to learn. The vast majority of experiments don’t affect metrics positively – but each one takes you closer to an eventual solution.

Children Don’t Worry About Being Wrong

Young children don’t get embarrassed if they get something wrong. As we grow up, however, we become more conservative and worry more about what people think. It is a product manager’s role to embrace the uncertainty that comes from failure so we can continue to improve our products.

Break Things Mindfully

You don’t want your experiments to create chaos. You want to create a system to explore how to get to your vision through testing your riskiest assumptions, so you should frame your thinking as a hypothesis. In short, you need to ask the right questions in the right way.

Good Product Managers are Curious

While product managers might not be specialists, good ones usually want to find out about the world around them. This might be their users, the world or their products. We all have the capacity to ask insightful questions, and a product manager’s role is to create an environment where the right questions can be asked without fear of failure or criticism.

Your Hypotheses Must be Based on Insights, Testable and Falsifiable

If we’re unable to prove that our tests are incorrect, then we can’t work out the cause and effect. We need to design our hypotheses so that the false situation can be found easily. Below is a structure that does this:

Based on [quant / qual insight,] we predict that [product change] will cause [impact]

By using this structure to our hypotheses we can base our tests on particular pieces of information and work out what is actually going on. We have a better chance of asking the right question in the right way.

Hypotheses Aren’t Good Enough by Themselves

No matter how good our experiments are, if we don’t consider the human aspect then they can become invalidated. The designers of any system are going to have a heavy confirmation bias to prove their ideas correct. They can even use an approach called HARKing – Hypothesising After the Results are Known – to redesign experiments once they have seen what happens in the real world…

Enthusiastic Scepticism for the win

In order to work out will work more quickly, you need to embrace the approach of proving yourself wrong. Focus on a single metric that’s important to your product and test using it over and over again. Hold yourself and your team accountable to finding out what affects it and what doesn’t.

Experiments Don’t Have to be big Feature Bets

Copy is one of the most important things to test for Skyscanner. By making simple text changes such as changing “Book” to “Select”, Skyscanner has seen double-digit improvements in conversion. These smaller iterations can make big differences when combined.

Design like you’re right, test like you’re wrong.

Find out more at http://www.experimentationhub.com/

James Gadsby Peet

About

James Gadsby Peet

I've been in the digital industry for over 10 years and have worked across small and large charities, as well as my own freelance projects. I am now Director of Digital at the sector leading creative agency William Joseph. Having been at Cancer Research UK for the last four years, I have been able to work on high volume, high profile campaigns such as Race for Life, Dryathlon and the much-quoted #nomakeupselfie. In that time I helped drive forward the charity’s digital product and marketing capabilities to be some of the most respected and successful in the sector. I'm always excited to work with other organisations, share expertise and swap cat gifs. Give me a shout!