Using Experimentation to Drive Product – Stephen Pavlovich (CEO of "Product people - Product managers, product designers, UX designers, UX researchers, Business analysts, developers, makers & entrepreneurs March 03 2017 True Experimentation, Testing, Ux Research, Mind the Product Mind the Product Ltd 636 Using Experimentation to Drive Product Product Management 2.544

Using Experimentation to Drive Product – Stephen Pavlovich (CEO of


Stephen Pavlovich, CEO of, talks to ProductTank London about Using Experimentation to Drive Product

Why aren’t people converting and what experiments can you run to try and fix that?

The aim of any experiment should be to better understand your users and apply those insights to your product. This does not have to be limited to your landing pages, sign up forms and eCommerce journeys – you can take the same principles and use them to improve your entire user experience. This can affect everything from product pricing, optimising lifetime value and even product functionality.

Experimentation can give you data you can’t get anywhere else

SaaS products often differentiate their packages by features. A free version will get you 2 / 5 capabilities and the Enterprise plan will give you the full suite plus added support. The issue here is that people don’t know what functionality they need until they have already signed up. ran a test version of a product where they created the levels by usage rather than functionality. So everyone got the same product, but the more you paid the more you could use it. This approach resulted in a 100% increase in revenue vs the more established method.

Insight into such behaviour is difficult through qualitative methods because people don’t know how they’d react in the situation you’re describing. Experimentation allows you to actually see people in the context you’re looking to optimise. The participants don’t even realise they’re in a test and so biases that invalidate the outcomes can’t be established.

You can find out things quicker through experimentation than anywhere else

In order to run a valid test you don’t need to implement a full version of the product or feature you’re looking for insight into. You can present a simplified version or perhaps even a link to something that looks like the feature and see how many people will click on it to test appetite. You can then make your roadmap or prioritisation decisions based on some data rather than just instinct. These could be looked at as ‘minimum viable tests’ which won’t show you all the aspects of the decisions you’re looking at, but will give you enough information to move forward.

Success is a measure of the choices we make. Experimentation let’s us peek at the result – in digital or physical.

If you are a company which makes physical products, then your supply chain will likely be highly complex. Making choices which have an impact on that chain will be high risk with the potential for an expensive impact if you get them wrong. In this situation using digital experiments, can give you an insight into whether it’s worth making the more expensive physical change.

For example, you might test whether it’s worth manufacturing a whole new product by offering it in the digital store first and seeing appetite – with a sold out banner later on in the process. This is obviously a poor user experience for the small number that it effects in the short term, but will result in a better product over the long term. For the short term cost in user experience, you are able to better direct your resources to what the majority of your users really care about.

Again the people in these tests are not having to vocalise or rationalise their behaviour as they would do in a focus group or interview – they are simply voting with their clicks. This results highly actionable data.

Use a hypothesis framework to give you an ‘unfair advantage’ against your competitors

The same framework that can apply for physical tests, can be used to manage your digital experiments or even your SEO programme variations.