Using Experimentation to Drive Product – Stephen Pavlovich (CEO of


Stephen Pavlovich, CEO of, talks to ProductTank London about Using Experimentation to Drive Product

Why aren’t people converting and what experiments can you run to try and fix that?

The aim of any experiment should be to better understand your users and apply those insights to your product. This does not have to be limited to your landing pages, sign up forms and eCommerce journeys – you can take the same principles and use them to improve your entire user experience. This can affect everything from product pricing, optimising lifetime value and even product functionality.

Experimentation can give you data you can’t get anywhere else

SaaS products often differentiate their packages by features. A free version will get you 2 / 5 capabilities and the Enterprise plan will give you the full suite plus added support. The issue here is that people don’t know what functionality they need until they have already signed up. ran a test version of a product where they created the levels by usage rather than functionality. So everyone got the same product, but the more you paid the more you could use it. This approach resulted in a 100% increase in revenue vs the more established method.

Insight into such behaviour is difficult through qualitative methods because people don’t know how they’d react in the situation you’re describing. Experimentation allows you to actually see people in the context you’re looking to optimise. The participants don’t even realise they’re in a test and so biases that invalidate the outcomes can’t be established.

You can find out things quicker through experimentation than anywhere else

In order to run a valid test you don’t need to implement a full version of the product or feature you’re looking for insight into. You can present a simplified version or perhaps even a link to something that looks like the feature and see how many people will click on it to test appetite. You can then make your roadmap or prioritisation decisions based on some data rather than just instinct. These could be looked at as ‘minimum viable tests’ which won’t show you all the aspects of the decisions you’re looking at, but will give you enough information to move forward.

Success is a measure of the choices we make. Experimentation let’s us peek at the result – in digital or physical.

If you are a company which makes physical products, then your supply chain will likely be highly complex. Making choices which have an impact on that chain will be high risk with the potential for an expensive impact if you get them wrong. In this situation using digital experiments, can give you an insight into whether it’s worth making the more expensive physical change.

For example, you might test whether it’s worth manufacturing a whole new product by offering it in the digital store first and seeing appetite – with a sold out banner later on in the process. This is obviously a poor user experience for the small number that it effects in the short term, but will result in a better product over the long term. For the short term cost in user experience, you are able to better direct your resources to what the majority of your users really care about.

Again the people in these tests are not having to vocalise or rationalise their behaviour as they would do in a focus group or interview – they are simply voting with their clicks. This results highly actionable data.

Use a hypothesis framework to give you an ‘unfair advantage’ against your competitors

The same framework that can apply for physical tests, can be used to manage your digital experiments or even your SEO programme variations.

James Gadsby Peet


I've been in the digital industry for over 10 years and have worked across small and large charities, as well as my own freelance projects. I am now Director of Digital at the sector leading creative agency William Joseph. Having been at Cancer Research UK for the last four years, I have been able to work on high volume, high profile campaigns such as Race for Life, Dryathlon and the much-quoted #nomakeupselfie. In that time I helped drive forward the charity’s digital product and marketing capabilities to be some of the most respected and successful in the sector. I'm always excited to work with other organisations, share expertise and swap cat gifs. Give me a shout!


Read more about Video ...

Building Awesome Product Teams by Martin Eriksson