Once upon a time, I met a product manager of a company who was leading a freemium model product, with some success. He regularly interviewed users to work out their needs, he gathered and analyzed usage data of the product to extract new requirements, and he defined new features with a high level of detail.
A Shiny new Feature
One day, he devised a new feature that promised to increase the free-to-paid users conversion rate by 5%. It was not a naive and under-researched wish. He collected loads of usage data, analyzed it and found a potential improvement to the user interface of the product that would, according to his calculations, lead to a substantial increase in the conversion rate, and by turn the company’s revenues.
The feature was developed on schedule, and successfully launched to a controlled group of users. The product manager measured the results, and astonishingly, the conversion increased by nearly 20%. This was a great deal better than anyone had expected, so it was rolled out to the entire user base the next day. Conversion to paid started to grow in an incredible pace, leading a proportional growth in revenues than made everyone in the company proud of the work that had been done.
But, the celebrations didn’t last long. Dozens of emails arrived with customer support, complaining that the company was forcing them to become premium users overnight. What was happening? No one was forcing them to pay in order to continue using the service.
The product manager did not understand. He started to go deep into the usage data for the new feature and found out that hundreds of users were signing out of the service. He couldn’t believe it. He scheduled a few calls and video conferences with users who had complained or stopped using the service, and finally he understood what was causing this exodus.
Misunderstanding at the User Interface
The user interface copy for the new call-to-action had been misunderstood by most users, it made them believe they had to pay or the service would no longer be available to them. Of course, this led users to choose between two options: pay or leave. This explained both results, the huge increase in conversion and the exit of so many users.
It was a terrible mistake. Nothing on earth would have made the product manager believe that the 20% conversion rate growth he saw with the control group would lead to such a catastrophic result when launching the feature for everybody.
Now, if you’ve reached this point of the article, you deserve some honesty from me.
I was that product manager.
With hindsight, I see there were several measures that I could have taken to avoid the error (I could have tested the new UI on real users even before passing it to the development team, gone deeper in the analysis I did on the control group, and so on).
Unfortunately, I let myself get so carried away by the excitement of growing revenues that I missed umpteen checks that would had avoided the tragedy.
But, a few days later, rather than continuing to blame myself for the mistake, I assumed that this kind of thing can occur – stuff happens. I just had to move on, keep working hard and, believe it or not, keep making mistakes.
Developing new products and making decisions around them implies taking risks. And things can go wrong. What is key is how we react to them and how we overcome the difficulties we find on a daily basis.
Making mistakes is not a big issue. The key thing here is to avoid making the same mistakes again. If we do this it means we are learning from them.