,
Artificial Intelligence (AI) & Machine Learning (ML)
SEP 19, 2024

LLM workflows for product managers: 3 key takeaways (Niloufar Salehi, Assistant Professor at UC Berkeley) – ProductTank SF

Share on

As a product manager, you're likely no stranger to dealing with complex system messes and the challenges that come with them. With the increasing presence of Artificial Intelligence (AI) in product management, it's essential to understand its impact on managing these messes and leveraging language models effectively. 

At ProductTank SF this year, Niloufar Salehi, Assistant Professor at UC Berkeley, dived into the world of LLM workflows and explored practical solutions for product managers.

Approximately 80% of data in organizations is considered "messy," making it difficult for product managers to navigate and use. Without a thorough understanding of data structure, model output can lack coherence and meaning. 

Creating a testing framework to evaluate model effectiveness is also a challenge due to the non-deterministic nature of messy data. Furthermore, end-users often evaluate language models based on subjective criteria, such as vibes and feelings, which makes benchmarking a complex task.

Veracity and trust are crucial components of high-confidence language models. Product managers must find a balance between technical understanding and practicality. Niloufar notes her experiences dealing with the potential consequences of translation errors in healthcare: using Google Translate when dealing with patients who only speak Mandarin.  Whilst doing research at UCSF, her team found that 8% of the sentences that were created through Google Translate, were incorrect, and some of those incorrect sentences resulted in causing clinical harm to patients. This example highlights the need for next-generation tools to account for unpredictable language translation.

With LLMs evidently being unreliable and non-deterministic, do they do really well that product managers can capitalise on? Niloufar believes that data, insights and patterns can become revolutionary with AI in addressing the problems for product managers:

Artificial Intelligence (AI) & Machine Learning (ML)

The future of product management: Insights from ProductTank San Francisco

Product lessons learned making early moves with AI in media: Lindsey Jayne (CPO, Financial Times)

50:43

A year with ChatGPT and product innovation: Navigating the AI landscape

24:28

How to keep your head about generative AI (when everyone is losing theirs) by Claire Woodcock

43:50

What we get wrong about technology by Tim Harford

Product management in the age of ChatGPT by Yana Welinder

How Canva uses AI-powered features to drive PLG

01:02:33

The role of an AI product manager by Hiroki Nakamura

20:17

A gentle introduction to AI in product by Rand Hindi

Recommended

43:50

What we get wrong about technology by Tim Harford

01:02:33

The role of an AI product manager by Hiroki Nakamura

20:17

A gentle introduction to AI in product by Rand Hindi

LLM workflows for product managers: 3 key takeaways (Niloufar Salehi, Assistant Professor at UC Berkeley) – ProductTank SF