Video: How to love A/B and Multivariate Testing

Running a multivariate test or even a simple AB test can be daunting. With each variation, there are additional tests to be run and measured, and it can all get overwhelming very quickly.  At ProductTank October, Alex Smith, the Site Development Manager at Charles Tyrwhitt, showed us what he measured and why, and talked about how he came to love running MVT and AB tests to learn more.

Alex talked about the A/B and multivariate testing program at Charles Tyrwhitt, outlining what they measure at and how the test results feed back into optimisation of the website.

He explained a simple formula for figuring out what to test, comparing traffic and expected benefit.  In his case, this left key pages like product listings and product pages, as well as the early and last steps of the checkout flow, to be the most ripe for optimisation through MVT and A/B testing.

Alex outlined the steps taken in the testing process, from benchmarking existing pages through to developing and testing your hypothesis, through to deploying changes based on the experiment results. He also outlined a number of simple ways to define success based on the results of an experiment.

As a result of this A/B and multivariate testing program that was put in place, Alex was able to show how it helped him get closer to the customer, give creativity room to breathe, removed some of the uncertainty from the development process, and helped to bring departments together.