post

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Successful product management in companies where it’s least understood

Measuring critical user journeys – Javier Andrés Bargas-Avila (UX Director, Google)

A new way to look at applied experiments

Product survey findings: Only 15% of users are embracing AI features 

Building high-performing product teams: The PDLC approach

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Guest Post

Overcoming the 20% feature usage challenge

Product Management

Three product leaders to learn from at #mtpcon inside Pendomonium

Laure Parsons is Senior User Researcher at Joya Communications, makers of the popular Marco Polo App. Previously she was Senior Product Manager at Notion. Laure has spoken at SXSW, Grace Hopper Celebration, ProductTank, and Optimal Workshop Roadshow, among others. She is a neuroscience geek and endlessly curious about what makes people do what they do. Find her on Twitter and Medium @laurex.

Laure Parsons

<p>I spoke to Lean Startup coach Tristan Kromer at Lean Startup Week earlier this month, where he presented a workshop about a “lean experiments toolbox”. Lean Startup Week is a conference for teams and leaders interested in applying a Lean Startup methodology to their product and Tristan helped to organise the Lean Unconference that kicked off the week’s activities. He is also editor and primary author of the <a href="http://www.trikro.com/downloads/playbook">Real Startup Book</a> and runs lean consulting firm TriKro LLC.</p>
<p>The product I work on, <a href="http://usenotion.com/">Notion</a>, is designed to help teams communicate and collaborate on their data so they can align everyone around their goals and experiments. I wanted to find out how Tristan approaches the use of data to guide product process.</p>
<h2>Designing your hypothesis</h2>
<p>Tristan recently published a blog which discusses the idea of ”<a href="https://grasshopperherder.com/assumption-vs-hypothesis-to-the-death/">assumptions vs. hypotheses</a>.” He believes that when just starting, our biggest challenge is identify our own assumptions rather than to build an MVP. I’m interested in <a href="http://www.slideshare.net/LaureParsons/overcome-your-mind-the-psychology-of-better-product-decisions">how cognitive bias can interfere</a> with product decisions, so I asked Tristan how teams can determine if their ideas are being posed as real foundations for experimentation. “An assumption is something accepted as true without question. A hypothesis is a proposed explanation or prediction that is a starting point for experimentation,” he says. “Assumptions are fine if they are made consciously. For example, I have an assumption that Javascript is a thriving programming language. I don&#8217;t really feel like testing that assumption and it&#8217;s very low risk, so I&#8217;ll leave it as an assumption I take for granted.”</p>
<p>However: “If we have an assumption such as ‘Everyone will buy our product’, that&#8217;s obviously very dangerous and probably not true. You can tell the difference because an assumption is usually quite vague and goes unchallenged as the truth. A hypothesis must be very specific, should make a falsifiable prediction, and should be discussed in the context of an experiment.”</p>
<h2>The metrics that matter</h2>
<p>Experiments generate data for a team to decide whether to move forward or to change direction. What does he feel are the most important metrics to inform these decisions? Below are four of the most important metrics in his practice:-</p>
<ul>
<li><b>Iteration Velocity</b> &#8211; How quickly is the team running experiments? (see more <a href="https://grasshopperherder.com/the-complete-team/">here</a>)</li>
<li><b>Knowledge / Experiment Ratio</b> &#8211; What % of those experiments are resulting in knowledge?</li>
<li><b>Team Morale</b> &#8211; How does the team get along and work together from a qualitative standpoint?</li>
<li><b>Compass</b> &#8211; Does the team have a product dashboard with actionable metrics?</li>
</ul>
<p>The Lean mantra is &#8220;fail fast&#8221;, but this can feel uncomfortable to many product people. The principle is rooted in the assertion that it is better to know if a product hypothesis fails to meet customer needs, so you avoid making things that don’t add value. Tristan comments: “The best way to avoid failure is to stop defining an invalid hypothesis as failure. It&#8217;s not failure, it&#8217;s learning. The only way we truly fail as a startup is if we&#8217;re not learning.”</p>
<h2>Does Lean really work?</h2>
<p>Tristan cites AirBnB as a true lean success story. As detailed in <a href="https://growthhackers.com/growth-studies/airbnb">this GrowthHackers case study</a>, AirBnB endured two years of early-stage development without seeing much traction. Like many startup founders, Brian Chesky and Joe Gebbia set out to solve a problem for themselves (they were struggling to pay the rent on their San Francisco apartment, and decided to engineer a better solution). But early on, customers were reluctant, as were some high-profile investors.</p>
<p>By using lean principles and relentless hustle, the company began to run experiments to increase confidence for reluctant guests and hosts. In one of their most famous experiments, AirBnB paid professional photographers to visit hosts and make listings more enticing. Customers responded and listings with the professional photos performed 2.5 times better. Says Tristan: “AirBnB is one of the most impressive success stories I&#8217;ve seen and the way they implement lean thinking consistently and even at scale is truly impressive.”</p>
<p>What then are some of the most successful lean experiments Tristan has seen? While many of the best results are protected by NDAs, one great example he has worked with is the now-shuttered music startup <a href="http://tentune.me/">tentune.me</a>.  He says: “They invalidated their business model and iterated incredibly quickly with paper prototypes in a coffee shop, but it doesn&#8217;t make for a sexy lean startup story because they ultimately decided the business was a bad idea and shut it down. I&#8217;m more impressed with people who have the courage to shut down bad ideas than people who were lucky enough to succeed with their first experiment.”</p>
<h2>Start at Learn</h2>
<p>“Build-Measure-Learn” may be the most famous of the Lean Startup mottos, but many teams struggle with how to implement experiments and data tracking in their organisation. What is Tristan’s advice is for teams who  want to start using lean principles? “Start at Learn; we shouldn&#8217;t start at Build,” he says. “We should start by asking, ‘What do we need to learn? What&#8217;s the riskiest thing about this business?’. Once we have that learning goal, we can go backwards to measure and figure out exactly what data would help us learn or answer a specific question. They we go to Build and build an experiment to help us acquire that data. So we go through the loop backwards before we go forward.”</p>
<p>Experiments can sometimes be subject to confirmation bias by the sponsor or proposer of the project or feature, or if the proposer has already decided they’ve come up with the right solution, it can be hard to invalidate. Tristan has this advice on combatting this issue: “The most important thing with buy-in on experiments is to get agreement on the fail conditions before running the experiment.”</p>
<p>He adds that a  good lean team will run an experiment to try and (in)validate the idea. But if the sponsor hasn&#8217;t agreed to the fail condition, their ego and reality distortion field will likely get in the way of accepting any negative results. “If we work with the sponsor as part of the team to agree on the criteria that would invalidate the hypothesis before the experiment, it&#8217;s very, very hard to ignore the data. It forces the sponsor into a position where they have to disagree with themselves in order to disagree with the results. That&#8217;s a cognitive dissonance that most people will try to avoid.”</p>
<p>I’m looking forward to sharing more takeaways from Lean Startup Week. For more insights into running lean experiments, Tristan’s recent article, “<a href="https://grasshopperherder.com/what-type-of-lean-startup-experiment-should-i-run/">What type of Lean Startup Experiment Should I Run?</a>” is worth a look.</p>
<div class="g g-21"><div class="g-single a-170"><a class="gofollow" data-track="MTcwLDIxLDEsNjA=" href="https://www.pendo.io/pendomonium/"><img decoding="async" src="https://www.mindtheproduct.com/wp-content/uploads/2024/06/MTP-Footer_ad.png" /></a></div></div>

Lean experiments: Tristan Kromer on better product process

Laure Parsons

Comments

Read next