Lean experiments: Tristan Kromer on better product process "Product people - Product managers, product designers, UX designers, UX researchers, Business analysts, developers, makers & entrepreneurs 7 June 2017 True Lean Startup, Product Development, Strategy, Mind the Product Mind the Product Ltd 1100 Product Management 4.4
· 5 minute read

Lean experiments: Tristan Kromer on better product process

I spoke to Lean Startup coach Tristan Kromer at Lean Startup Week earlier this month, where he presented a workshop about a “lean experiments toolbox”. Lean Startup Week is a conference for teams and leaders interested in applying a Lean Startup methodology to their product and Tristan helped to organise the Lean Unconference that kicked off the week’s activities. He is also editor and primary author of the Real Startup Book and runs lean consulting firm TriKro LLC.

The product I work on, Notion, is designed to help teams communicate and collaborate on their data so they can align everyone around their goals and experiments. I wanted to find out how Tristan approaches the use of data to guide product process.

Designing your hypothesis

Tristan recently published a blog which discusses the idea of ”assumptions vs. hypotheses.” He believes that when just starting, our biggest challenge is identify our own assumptions rather than to build an MVP. I’m interested in how cognitive bias can interfere with product decisions, so I asked Tristan how teams can determine if their ideas are being posed as real foundations for experimentation. “An assumption is something accepted as true without question. A hypothesis is a proposed explanation or prediction that is a starting point for experimentation,” he says. “Assumptions are fine if they are made consciously. For example, I have an assumption that Javascript is a thriving programming language. I don’t really feel like testing that assumption and it’s very low risk, so I’ll leave it as an assumption I take for granted.”

However: “If we have an assumption such as ‘Everyone will buy our product’, that’s obviously very dangerous and probably not true. You can tell the difference because an assumption is usually quite vague and goes unchallenged as the truth. A hypothesis must be very specific, should make a falsifiable prediction, and should be discussed in the context of an experiment.”

The metrics that matter

Experiments generate data for a team to decide whether to move forward or to change direction. What does he feel are the most important metrics to inform these decisions? Below are four of the most important metrics in his practice:-

  • Iteration Velocity – How quickly is the team running experiments? (see more here)
  • Knowledge / Experiment Ratio – What % of those experiments are resulting in knowledge?
  • Team Morale – How does the team get along and work together from a qualitative standpoint?
  • Compass – Does the team have a product dashboard with actionable metrics?

The Lean mantra is “fail fast”, but this can feel uncomfortable to many product people. The principle is rooted in the assertion that it is better to know if a product hypothesis fails to meet customer needs, so you avoid making things that don’t add value. Tristan comments: “The best way to avoid failure is to stop defining an invalid hypothesis as failure. It’s not failure, it’s learning. The only way we truly fail as a startup is if we’re not learning.”

Does Lean really work?

Tristan cites AirBnB as a true lean success story. As detailed in this GrowthHackers case study, AirBnB endured two years of early-stage development without seeing much traction. Like many startup founders, Brian Chesky and Joe Gebbia set out to solve a problem for themselves (they were struggling to pay the rent on their San Francisco apartment, and decided to engineer a better solution). But early on, customers were reluctant, as were some high-profile investors.

By using lean principles and relentless hustle, the company began to run experiments to increase confidence for reluctant guests and hosts. In one of their most famous experiments, AirBnB paid professional photographers to visit hosts and make listings more enticing. Customers responded and listings with the professional photos performed 2.5 times better. Says Tristan: “AirBnB is one of the most impressive success stories I’ve seen and the way they implement lean thinking consistently and even at scale is truly impressive.”

What then are some of the most successful lean experiments Tristan has seen? While many of the best results are protected by NDAs, one great example he has worked with is the now-shuttered music startup tentune.me.  He says: “They invalidated their business model and iterated incredibly quickly with paper prototypes in a coffee shop, but it doesn’t make for a sexy lean startup story because they ultimately decided the business was a bad idea and shut it down. I’m more impressed with people who have the courage to shut down bad ideas than people who were lucky enough to succeed with their first experiment.”

Start at Learn

“Build-Measure-Learn” may be the most famous of the Lean Startup mottos, but many teams struggle with how to implement experiments and data tracking in their organisation. What is Tristan’s advice is for teams who  want to start using lean principles? “Start at Learn; we shouldn’t start at Build,” he says. “We should start by asking, ‘What do we need to learn? What’s the riskiest thing about this business?’. Once we have that learning goal, we can go backwards to measure and figure out exactly what data would help us learn or answer a specific question. They we go to Build and build an experiment to help us acquire that data. So we go through the loop backwards before we go forward.”

Experiments can sometimes be subject to confirmation bias by the sponsor or proposer of the project or feature, or if the proposer has already decided they’ve come up with the right solution, it can be hard to invalidate. Tristan has this advice on combatting this issue: “The most important thing with buy-in on experiments is to get agreement on the fail conditions before running the experiment.”

He adds that a  good lean team will run an experiment to try and (in)validate the idea. But if the sponsor hasn’t agreed to the fail condition, their ego and reality distortion field will likely get in the way of accepting any negative results. “If we work with the sponsor as part of the team to agree on the criteria that would invalidate the hypothesis before the experiment, it’s very, very hard to ignore the data. It forces the sponsor into a position where they have to disagree with themselves in order to disagree with the results. That’s a cognitive dissonance that most people will try to avoid.”

I’m looking forward to sharing more takeaways from Lean Startup Week. For more insights into running lean experiments, Tristan’s recent article, “What type of Lean Startup Experiment Should I Run?” is worth a look.

Comments 1


You point out that the Build Measure Learn loop should start at Learn, not Build. I agree that starting at Build contradicts Lean thinking. However, starting at Learn doesn’t really make much sense, since you haven’t done anything yet that you can learn from.

I prefer a four-stage loop that starts with “Hypothesize” followed by Build Measure Learn. This is much closer to reality (and much easier to justify and explain to founders). In addition, it corresponds to the Method of Science, which is what Lean Startup is trying to emulate.

“Build Measure Learn” is the second-worst misnomer in the startup ecosystem. (The worst misnomer is “Minimum Viable Product”, which is usually not a product at all.) Both lead to a lot of confusion among founders.

Best Regards

Join the community

Sign up for free to share your thoughts

About the author