I’m currently on the brink of launching a major product update – one that will affect many of our high value clients. And yes, it’s stressful. The questions swirling around in my head include:
- Does the functionality work in production?
- How do the users respond?
- How does this impact operational cost?
In other words, do we have the holy trinity of Feasibility / Desirability / Viability? This article outlines how to manage these questions, from a product manager who’s been there.
The Stars Align: Launching Heatmap
One of my team’s big projects this year is to do a V2 of one of our platform’s features, called Heatmap, an insights-based tool clients use to gather info about how people move through their website. This is a big task indeed, since the tool requires a lot of storage space and its tech stack is different from our main platform. In addition, many of our clients were already using and very invested in the V1. They were both expecting a killer V2, and also wouldn’t be too happy with any impromptu glitches or bugs.
As fate would have it, last year we launched a brand new product that was designed to make the lives of Product Managers like me a whole lot easier. It’s a feature management and product optimization platform, and includes progressive rollout and personalization capabilities based – in part – on simple feature flags, wrapped in a product-manager-friendly SaaS console.
Whether you happen to be using our product, one of the other feature management solutions out there, or you’ve built (or can build) your own in-house solution, using some kind of progressive rollout tactic for a major release like this one is one of the best ways I’ve come across to manage the inevitable stress of putting a new, high value feature into the world. Let’s walk through why:
Perks of Progressive Rollout
For a product manager on the brink of launching a total overhaul of a popular feature in the hands of numerous clients, there are both external-facing, and internal-facing advantages to using a feature management platform or process.
First and foremost, a canary launch or progressive rollout approach lets you test the waters of a new version with a restricted set of clients. If any unanticipated issues come to light, you can pause the full launch, fix the issues, and then smoothly move ahead. The famous, uncouple code delivery from feature release, approach.
The other big advantage? Anticipating costs.
In our case, our Heatmap tool is based on Google Cloud and, as I’m sure the reader is aware, each API call costs money. The problem is, it’s impossible for us to foresee exactly how many calls – and therefore, how much of a budget – will be required as we develop the V2. Of course, Google Cloud (and Amazon, and Microsoft) offer ways to simulate and anticipate costs. And yes, we do make use of them. But the final invoice always comes out more bloated than what we had anticipated.
For me, the unforeseen costs that blow your provisional budgets sky high at the end of the month are almost as bad as the night sweats that come from an unforeseen bug! Monitoring our costs during a progressive rollout – and immediately putting the launch on pause if those costs spike – is a phenomenal level of control that I absolutely wanted to get in on. For this launch, being sure our cloud infrastructure could scale reasonably was key, and I’m sure this is the case for many other product managers in my shoes.
Progressive Rollouts in B2C and B2B – Feature Flags vs Audience Segments
Now, a quick note for the reader – if you’re working for a B2B company like I am, a progressive rollout technique like using feature flagging is probably the most relevant for you. Especially if you’ve built up strong relationships with individual clients, to the point where you can call them up, they know who you are, and they are willing to chat. In that kind of situation, all you need are a handful of willing beta testers you can turn a feature ‘on’ for. You iterate on their feedback, and then come back with a general release version.
In B2C companies, you’re unlikely to have this type of relationship with clients. Instead, you most likely have masses of anonymous users to work with. In that situation, a progressive rollout based on smart KPIs, with some kind of data-driven audience segmentation, will be a better way of gathering user feedback. If you launch a new feature to 10% of your client base and notice revenue or engagement takes a dip, you can pause the release and investigate. Of course, you need the relevant feature management platform to make that happen.
Basically, taking a moment to reflect on how to best use a progressive rollout approach, depending on your user base and market context, will help you get the most out of this technique.
An Ideal Feature Release
As a product manager, I want to be able to anticipate and control as much as possible about a release. So we found a way to minimize risk and maximize results (and I’d recommend these steps for any other product manager in my shoes):
Step one: Build your Sample Audience.
The whole point of a progressive rollout is to target a subset of your audience as a test case. You need to think about sample size, sampling biases etc… In a B2B context, (as I explain below), you can work closely with a Customer Success or Account Manager to choose strategic clients that make sense.
In B2C, decide whether you want a representative sample of your audience, or if you want to target a specific segment (e.g. low-risk, high-feedback, whatever). Find KPIs to identify them, and target based on that.
In our case, and with the help of our Customer Success and Key Account Manager teams, I shortlisted around 10 clients that showed interest for an early adopter launch of our V2 heatmap. I picked clients that often use the feature (so they’ll be more likely to provide relevant feedback), that showed they’re open to giving this kind of input during a beta test. I also ensured the sample was diverse, intentionally mixing up the industries and sizes of these clients. (At this stage we don’t necessarily want a representative sample of our client base, we want a maximum of diversity.)
Then, at a date we chose as a team, activated them using our feature management platform. This was done manually via a simple feature flag, but the point is that we had a consistent “soft launch” date for everyone.
Step two: Decide on a Sensible Window of Time to Measure Impact With Confidence.
Think about your customers’ usage patterns, or about how often Account Managers contact clients. Loop in marketing or communications to factor in their plans (or any big industry events).
Decide what would be a window of time (or different windows for different sample segments) that could give you confidence in the quality of your release, and communicate that with stakeholders.
If we come back to our heatmap example, we did just that – we waited a predetermined amount of time and monitored to see if any bugs came up. We checked in regarding any client feedback. As product managers, this kind of “live testing” is key for a continuous optimization approach – and it was important that we took the time and resources to iterate based on this first round of real user feedback.
As mentioned above, we also looped in our marketing and communication teams here to decide on what date made the most sense to announce a general release. When are we sending out the press release, the email blast, the social media ads?
This is in fact the stage we are currently in. The next steps are what we have planned to make sure the rest of the release goes smoothly (and it is crucial that walk through these steps ahead of time, so that you have a full plan in place before you begin your progressive rollout)…
Step three: Identify Pass / Fail Conditions, and Metrics to Watch:
You’re effectively running an experiment to determine if your release is feasible, desirable & viable.
As a team, determine what you expect will happen (based on your research, evidence, and models). Ideally, you should identify what results would force you to halt the release, investigate, make a fix, and try again. These goals should be based on conversations you’ve had with the various release stakeholders, and should be clearly communicated & understood so that everyone is aligned on what will either allow this release out the door or apply the brakes.
For us, that meant thinking about cloud costs – were they getting out of control? And also, was there consequential negative user feedback? If this is the case, I’ll use our feature management platform to pause the rollout, and then restart once the issues have been patched and we’re ready to continue the experiment.
Step four: Plan the Rest of Your Progressive Rollout:
Feature rollouts like this aren’t just one experiment – they’re a series of experiments of increasing scope, designed to gradually test & validate your product hypotheses, and build confidence that you’re moving in the right direction. At the start of the process, you should map out how you intend to roll your new release out to the rest of your users. You should consider:
- How you’d like to segment the rest of your users (in terms of risk & segment size)
- How long you think each segment should be able to use the new release before you’re confident enough to progress.
- How long is the complete rollout likely to take? Are there marketing plans or industry events you should be aware of? How might those impact your rollout?
In our case that means once we get the V2 Heatmap where we want it, we’ll use a progressive rollout approach to space out full feature activation over 15 days—again, another precaution against bugs, and especially skyrocketing costs (making sure IT infrastructure is scaling as it needs to be.)
This is a much more controlled, low-stress way of launching a much anticipated update.
Looking Forward
Hopefully, you’ll will have picked up a few tips on how to de-stress a big launch using progressive rollout techniques. Another lesson that we learned at AB Tasty that I think is applicable to any tech company? Drink your own champagne. It’s ok to start with a sip, but the sooner you take a full swig, the better!
When we first launched Flagship, we adopted it sparingly for our own releases. We needed time to build up our internal processes and incorporate a new tool. But after a few more releases, we got the hang of it. Now, it’s become an invaluable tool in our kit for releases big and small, and the value is evident.
