Six years ago at Skyscanner there was no experimentation: now, there might be 500 tests running at any one time. In this #mtpcon London talk, Skyscanner’s Principal Product Manager Rik Higham looks at how experimentation can be a strategic capability for an organisation.
First you should change your approach to failure – rather than look at failure as from something that didn’t work, consider that you’ve only failed if you failed to learn. The vast majority of experiments don’t affect metrics positively – but each one takes you closer to an eventual solution.
Children Don’t Worry About Being Wrong
Young children don’t get embarrassed if they get something wrong. As we grow up, however, we become more conservative and worry more about what people think. It is a product manager’s role to embrace the uncertainty that comes from failure so we can continue to improve our products.
Break Things Mindfully
You don’t want your experiments to create chaos. You want to create a system to explore how to get to your vision through testing your riskiest assumptions, so you should frame your thinking as a hypothesis. In short, you need to ask the right questions in the right way.
Good Product Managers are Curious
While product managers might not be specialists, good ones usually want to find out about the world around them. This might be their users, the world or their products. We all have the capacity to ask insightful questions, and a product manager’s role is to create an environment where the right questions can be asked without fear of failure or criticism.
Your Hypotheses Must be Based on Insights, Testable and Falsifiable
If we’re unable to prove that our tests are incorrect, then we can’t work out the cause and effect. We need to design our hypotheses so that the false situation can be found easily. Below is a structure that does this:
Based on [quant / qual insight,] we predict that [product change] will cause [impact]
By using this structure to our hypotheses we can base our tests on particular pieces of information and work out what is actually going on. We have a better chance of asking the right question in the right way.
Hypotheses Aren’t Good Enough by Themselves
No matter how good our experiments are, if we don’t consider the human aspect then they can become invalidated. The designers of any system are going to have a heavy confirmation bias to prove their ideas correct. They can even use an approach called HARKing – Hypothesising After the Results are Known – to redesign experiments once they have seen what happens in the real world…
Enthusiastic Scepticism for the win
In order to work out will work more quickly, you need to embrace the approach of proving yourself wrong. Focus on a single metric that’s important to your product and test using it over and over again. Hold yourself and your team accountable to finding out what affects it and what doesn’t.
Experiments Don’t Have to be big Feature Bets
Copy is one of the most important things to test for Skyscanner. By making simple text changes such as changing “Book” to “Select”, Skyscanner has seen double-digit improvements in conversion. These smaller iterations can make big differences when combined.
Design like you’re right, test like you’re wrong.
Find out more at http://www.experimentationhub.com/
