When you use an AI tool, how much time do you spend correcting it? And would your customers be willing to do the same?
Kendra Vant, Chief Product Officer at Tapi, joins Randy Silver to explore the “reliability layer”: the people, checks and engineering that make AI products dependable. She explains why enthusiastic AI users often provide that layer themselves, and why product teams can’t assume their customers will.
Kendra shares four questions to ask before building AI into your product, from who catches mistakes to what reliability costs at scale. They also discuss designing for failure, the technical literacy she expects from product managers, and why faster prototyping makes experienced judgement so valuable.
Chapters
- 00:00 Introducing Kendra Vant
- 02:51 The gap between an AI demo and a reliable product
- 06:12 Accuracy, consistency and the user as the reliability layer
- 13:31 Four questions to ask before building AI into your product
- 18:27 Sponsor: Jira Product Discovery
- 19:01 Planning for failure and protecting customer trust
- 24:06 Customisation, pricing and the cost of scaling
- 26:37 Do product managers need to learn to code?
- 31:40 How Tapi uses AI to experiment and build
- 33:25 Why small teams have an advantage
- 35:05 Product judgement and the experience gap
- 38:08 Using AI to maintain legacy code
- 39:41 Ask better questions, write down answers and test your thinking
- 43:11 Where to follow Kendra
Key takeaways
- Find the work your users are doing for the AI. Correcting answers, refining prompts and spotting mistakes can make a tool feel more reliable than it is. Consider whether your customers have the time, motivation and expertise to do that work.
- Make someone responsible for reliability. Establish who specifies, builds and operates the checks around your AI. An impressive demo can conceal the fact that nobody owns this work yet.
- Price the whole product. Human review, additional models, guardrails and ongoing maintenance all affect the cost of delivery. Check whether your reliability layer remains commercially viable as usage grows.
- Design for the moments when it fails. Saying a model is wrong 15% of the time prompts a different conversation from saying it is 85% accurate. Identify the consequences for customers and plan how the product will recover.
- Build your technical literacy. Kendra expects product managers to be comfortable interacting with Git and learning from the codebase. Understanding how software works helps you ask better questions and collaborate with engineers.
- Recognise the value of experience. Faster tools increase what teams can build, but recognising flawed suggestions still requires judgement. Kendra raises an open question: how will newer practitioners develop that judgement as the work changes?
- Use writing to sharpen your thinking. Break difficult problems into smaller questions, write down your answers and explain them to a colleague. The gaps often become clearer when you have to articulate your reasoning.
We're refreshing The Product Experience and want your input. Take our two-minute survey and help shape where the show goes next!
