Deep dive: Revolutionising product management with ChatGPT "Product people - Product managers, product designers, UX designers, UX researchers, Business analysts, developers, makers & entrepreneurs 6 November 2023 False ChatGPT, Guest Post, Premium Content, Mind the Product Mind the Product Ltd 1599 Revolutionizing product management with GPT: A Deep Dive into How AI Can Help Product Managers Succeed Product Management 6.396

Deep dive: Revolutionising product management with ChatGPT

This article is part of our AI Knowledge Hub, created with Pendo. For similar articles and even more free AI resources, visit the AI Knowledge Hub now.

This deep dive guest post written by Senior Global Product Manager, Jin Hu, delves into the opportunities that lie ahead for product managers with the arrival of ChatGPT and other advances in AI.

Hi there! As a product person in back office operations, I always try to find ways to improve efficiency, even if it means eventually eliminating some jobs (please don’t hate me, this is often a consequence of doing my job well). Lately, I’ve been really interested in exploring how GPT can help streamline my work process and improve the productivity of the products I’m responsible for.

A few days ago I came across a paper titled “GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models” by authors from OpenAI, OpenResearch, and the University of Pennsylvania. It’s been getting a lot of buzz, with several YouTubers discussing its findings. I highly recommend reading the paper or checking out some of the videos, I left some links at the end of this article.

I initiated a chat with GPT out of pure curiosity regarding a table mentioned in the paper (Screenshot below). This table describes the regression of occupation-level, human-annotated exposure to GPTs on skill importance for each skill in the O*NET Basic skills category, plus the programming skill.

To explain this to a 5-year-old, the higher the number in the table the more the skill is likely to be augmented/exposed by different levels of LLM involvement (𝛼, 𝛽, 𝜁. search the paper for more detail). Exposed here means “a measure of whether access to an LLM or LLM-powered system would reduce the time required for a human to perform a specific DWA or complete a task by at least 50 percent.

The conclusion made in the paper on this table is the same as you can see here.

The higher the number, the augmentation is stronger, and the negative number means the LLM could not be of assistance to augment the specific skill.

Now we have set the stage. Let’s continue to follow my journey.

I wanted to know how product managers’ day-to-day tasks mapped with the O*NET Basic Skills and find out how many of them required “science and critical thinking”. Here’s a table that GPT generated based on my requests:

Table 1. A list of generic product manager tasks mapped with O*NET Basic Skills also other skills needed identified by GPT. Each row is the skills required to complete the respective task successfully.

Please note, this is a list of tasks for an individual contributing product manager, not for people managers. Also, I am well aware this isn’t a comprehensive list of tasks nor the O*NET skills matched perfectly. I still decided to publish it as-is, given this isn’t the main point to make here and it’s good enough.

There were a few “errors” from GPT that I had to correct during the chat; otherwise, the result was very impressive. Here are three types of errors made:

  • No clear explanation on why a task can/can’t be assisted by GPT, e.g., GPT initially said it couldn’t help with user research. I argued otherwise. I broke down the research process into more specific tasks and pointed out which tasks could use GPT for productivity.
  • GPT picked up some more niche requirements for product managers and not necessarily applicable to all product managers, e.g., GPT initially suggested product managers are responsible for pricing.
  • GPT failed to pick up some generic product manager’s day-to-day tasks, e.g., stakeholder management.

From time to time, there are similar errors in one answer from GPT. When I asked it to correct the error, it normally also picked up the other ones and provided a better version of output.

Based on the statement from the paper and combined the table from GPT, I conclude more than half of the PM tasks won’t be exposed to GPT4 (current algorithm). Saying this, I wanted to know in more detail, exactly which are the tasks the GPT can or can not help with. After a while, our chat grew closer to what I was looking for. Here’s the result, a table listing generic product managers’ day-to-day tasks, whether GPT can be of assistance, and how.

Table 2. Continue using the same set of the generic product management tasks, based on the description of each task to see if GPT recognise this is something GPT can be of assistance, and in what way.

As you can see here, there is no single definitive “Yes” in the table, only some “Maybes.” You might also find that the “Maybe” in this table isn’t exactly the same tasks as the tasks that require Critical thinking and science as skills. I think two reasons, first is that even a big part of a product manager task requires Critical thinking and scientific research skills, but they still can be broken down into smaller parts that could be assisted by GPT.

Second, the context between Table 1 and Table 2 is different. Table 1 is about mapping between the O*NET Basic Skills and the product management tasks based on a statement from the paper. Whereas there is no pre-defined list or standard for GPT to reference on when we generated Table 2, all information is gathered based on what’s available for the current GPT version.

 

Until next time …

In conclusion, our conversation was a fascinating exploration of the essential skills of product managers and how GPT could assist with them. Here is a list of tasks you can try getting help from GPT. Some of them require GPT to have access to more context, eg. the detail and background of the program you’re working on, etc. Also, some of the work that could be done by GPT should in theory be further improved by apps built on top of AGI with specific instructions.

  1. Conduct market research: GPT can assist with this task by analyzing online forums, social media, and other sources of data to identify trends and patterns in consumer behaviour.
  2. Identify user needs and pain points: GPT can assist with this task by analyzing user feedback and identifying common themes and issues that need to be addressed.
  3. Create user stories: GPT can assist with this task by generating user stories based on user feedback and requirements. The premise is that GPT has access to all the context of the company and the program you are part of.
  4. Develop product positioning and messaging: GPT can assist with this task by analyzing market trends and consumer behaviour to develop effective product positioning and messaging.
  5. Create a Go-To-Market strategy: GPT can assist with this task by analyzing market trends and consumer behaviour to develop effective Go-To-Market strategies.
  6. Product marketing: GPT can assist with this task by generating marketing copy and content based on market trends and consumer behaviour.

While GPT could provide some assistance in certain areas, human input and decision-making were still crucial in ensuring the success of product management. It was clear that while AI has the potential to revolutionize the field of product management, it could not yet fully replace the skills and expertise of a human product manager.

Photo by Ray ZHUANG on Unsplash

Appendix- O*NET Basic Skills

Content

These are background structures needed to work with and acquire more specific skills in a variety of different domains.

  • Reading Comprehension — understanding written sentences and paragraphs in work-related documents.
  • Active Listening — giving full attention to what other people are saying, taking time to understand the points being made, asking questions as appropriate, and not interrupting at inappropriate times.
  • Writing — communicating effectively in writing as appropriate for the needs of the audience.
  • Speaking — talking to others to convey information effectively.
  • Mathematics — using mathematics to solve problems.
  • Science — using scientific rules and methods to solve problems.

Process

These are procedures that contribute to the more rapid acquisition of knowledge and skill across a variety of domains.

  • Critical Thinking — using logic and reasoning to identify the strengths and weaknesses of alternative solutions, conclusions, or approaches to problems.
  • Active Learning — understanding the implications of new information for both current and future problem-solving and decision-making.
  • Learning Strategies — selecting and using training/instructional methods and procedures appropriate for the situation when learning or teaching new things.
  • Monitoring — monitoring/assessing performance of yourself, other individuals, or organizations to make improvements or take corrective action.

Cross-Functional Skills

The paper also selected Programming from the list of cross-functional skills because of our prior knowledge about the models’ ability to code.

  • Programming — writing computer programs for various purposes.

Serious stuff- LLM Assistance And Research Statement

GPT-4 and ChatGPT were used for writing, coding, and formatting assistance in this project.

Additional research is required to investigate the wider implications of LLM progress, including their potential to enhance or replace human labour, the effect on job quality, inequality impacts, skill development, and other outcomes. By attempting to comprehend the abilities and potential consequences of LLMs on the workforce, policymakers and stakeholders can make better decisions to navigate the complex AI landscape and its role in shaping the future of work.

Reference and useful links

Comments 5

Great article. You’re spot on with this statement that some of the product tasks require GPT to have access to more context, eg. the detail and background of the program you’re working on.

This is exactly how and why we built PaceAI (askpace.ai) – a tool for helping growing Product Managers to generate ideas in seconds instead of hours or days. We built it to allow users to provide context or description of their product/project and let the AI generate some ideas that can then be improved.

Thanks for such an awesome article.

Hi Patrick,

First off, my apologies for the delay in responding to your comment. I had a slight issue with logging in, but all is sorted now.

Your comment is much appreciated! Your tool sounds like a fascinating application of AI, especially as it aims to reduce time spent on idea generation for Product Managers. I must say, I’m curious to know how it’s been received by users and the impact it has had so far.

Hi Jing, thank you for this piece. As a mid-level product manager, it’s easy to get carried away with all the buzz coming from the AI space.

How do you reckon that people like me (mid level PMs or newbies) stay ahead, or at least use AI for our benefit, particularly in the area of Conducting Market Research?

Just adding my little cent here – There are many ways to leverage AI as a growing PM.

There are many newsletters to stay up to date with news and trends, but I particularly like product prompts’s newsletter – (https://www.posts.productprompts.com/). Martin writes and covers all things AI but from a product management lens which is great.

To leverage AI for your benefit, check out PaceAI (askpace.ai) – an AI companion for growing Product Managers to generate ideas in seconds instead of hours or days.

Hi Christian,

Firstly, I apologize for the delay in responding to your insightful comment. I had a minor hiccup with logging in, but I’m now able to respond.

Your question really sparked my interest, and I found myself mulling over it for some time. As a mid-level product manager, it’s indeed essential to understand how to navigate and leverage the dynamic AI space, particularly in areas such as market research.

The thoughts and ideas your comment provoked were so significant that they inspired me to delve deeper into this subject and write another article. The piece, “From LaLa Land to The Matrix: Can AI replace product management?”(https://www.mindtheproduct.com/from-lala-land-to-the-matrix-can-ai-replace-product-management/) explores how product managers can use their unique expertise in conjunction with AI for innovation and growth.

I believe it provides some practical insights into the question you asked. I’d be more than happy if you took the time to read it and share your thoughts. Should you wish to further this discussion, please do not hesitate to let me know.

Join the community

Sign up for free to share your thoughts

This article is part of our AI Knowledge Hub, created with Pendo. For similar articles and even more free AI resources, visit the AI Knowledge Hub now. This deep dive guest post written by Senior Global Product Manager, Jin Hu, delves into the opportunities that lie ahead for product managers with the arrival of ChatGPT and other advances in AI. Hi there! As a product person in back office operations, I always try to find ways to improve efficiency, even if it means eventually eliminating some jobs (please don’t hate me, this is often a consequence of doing my job well). Lately, I’ve been really interested in exploring how GPT can help streamline my work process and improve the productivity of the products I’m responsible for. A few days ago I came across a paper titled “GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models” by authors from OpenAI, OpenResearch, and the University of Pennsylvania. It’s been getting a lot of buzz, with several YouTubers discussing its findings. I highly recommend reading the paper or checking out some of the videos, I left some links at the end of this article. I initiated a chat with GPT out of pure curiosity regarding a table mentioned in the paper (Screenshot below). This table describes the regression of occupation-level, human-annotated exposure to GPTs on skill importance for each skill in the O*NET Basic skills category, plus the programming skill. To explain this to a 5-year-old, the higher the number in the table the more the skill is likely to be augmented/exposed by different levels of LLM involvement (𝛼, 𝛽, 𝜁. search the paper for more detail). Exposed here means “a measure of whether access to an LLM or LLM-powered system would reduce the time required for a human to perform a specific DWA or complete a task by at least 50 percent. The conclusion made in the paper on this table is the same as you can see here. The higher the number, the augmentation is stronger, and the negative number means the LLM could not be of assistance to augment the specific skill. Now we have set the stage. Let’s continue to follow my journey. I wanted to know how product managers’ day-to-day tasks mapped with the O*NET Basic Skills and find out how many of them required “science and critical thinking”. Here’s a table that GPT generated based on my requests: Table 1. A list of generic product manager tasks mapped with O*NET Basic Skills also other skills needed identified by GPT. Each row is the skills required to complete the respective task successfully. Please note, this is a list of tasks for an individual contributing product manager, not for people managers. Also, I am well aware this isn’t a comprehensive list of tasks nor the O*NET skills matched perfectly. I still decided to publish it as-is, given this isn’t the main point to make here and it’s good enough. There were a few “errors” from GPT that I had to correct during the chat; otherwise, the result was very impressive. Here are three types of errors made:
  • No clear explanation on why a task can/can’t be assisted by GPT, e.g., GPT initially said it couldn’t help with user research. I argued otherwise. I broke down the research process into more specific tasks and pointed out which tasks could use GPT for productivity.
  • GPT picked up some more niche requirements for product managers and not necessarily applicable to all product managers, e.g., GPT initially suggested product managers are responsible for pricing.
  • GPT failed to pick up some generic product manager's day-to-day tasks, e.g., stakeholder management.
From time to time, there are similar errors in one answer from GPT. When I asked it to correct the error, it normally also picked up the other ones and provided a better version of output. Based on the statement from the paper and combined the table from GPT, I conclude more than half of the PM tasks won’t be exposed to GPT4 (current algorithm). Saying this, I wanted to know in more detail, exactly which are the tasks the GPT can or can not help with. After a while, our chat grew closer to what I was looking for. Here’s the result, a table listing generic product managers' day-to-day tasks, whether GPT can be of assistance, and how. Table 2. Continue using the same set of the generic product management tasks, based on the description of each task to see if GPT recognise this is something GPT can be of assistance, and in what way. As you can see here, there is no single definitive “Yes” in the table, only some “Maybes.” You might also find that the “Maybe” in this table isn’t exactly the same tasks as the tasks that require Critical thinking and science as skills. I think two reasons, first is that even a big part of a product manager task requires Critical thinking and scientific research skills, but they still can be broken down into smaller parts that could be assisted by GPT. Second, the context between Table 1 and Table 2 is different. Table 1 is about mapping between the O*NET Basic Skills and the product management tasks based on a statement from the paper. Whereas there is no pre-defined list or standard for GPT to reference on when we generated Table 2, all information is gathered based on what’s available for the current GPT version.  

Until next time …

In conclusion, our conversation was a fascinating exploration of the essential skills of product managers and how GPT could assist with them. Here is a list of tasks you can try getting help from GPT. Some of them require GPT to have access to more context, eg. the detail and background of the program you’re working on, etc. Also, some of the work that could be done by GPT should in theory be further improved by apps built on top of AGI with specific instructions.
  1. Conduct market research: GPT can assist with this task by analyzing online forums, social media, and other sources of data to identify trends and patterns in consumer behaviour.
  2. Identify user needs and pain points: GPT can assist with this task by analyzing user feedback and identifying common themes and issues that need to be addressed.
  3. Create user stories: GPT can assist with this task by generating user stories based on user feedback and requirements. The premise is that GPT has access to all the context of the company and the program you are part of.
  4. Develop product positioning and messaging: GPT can assist with this task by analyzing market trends and consumer behaviour to develop effective product positioning and messaging.
  5. Create a Go-To-Market strategy: GPT can assist with this task by analyzing market trends and consumer behaviour to develop effective Go-To-Market strategies.
  6. Product marketing: GPT can assist with this task by generating marketing copy and content based on market trends and consumer behaviour.
While GPT could provide some assistance in certain areas, human input and decision-making were still crucial in ensuring the success of product management. It was clear that while AI has the potential to revolutionize the field of product management, it could not yet fully replace the skills and expertise of a human product manager. [caption id="" align="alignnone" width="1067"] Photo by Ray ZHUANG on Unsplash[/caption]

Appendix- O*NET Basic Skills

Content

These are background structures needed to work with and acquire more specific skills in a variety of different domains.
  • Reading Comprehension — understanding written sentences and paragraphs in work-related documents.
  • Active Listening — giving full attention to what other people are saying, taking time to understand the points being made, asking questions as appropriate, and not interrupting at inappropriate times.
  • Writing — communicating effectively in writing as appropriate for the needs of the audience.
  • Speaking — talking to others to convey information effectively.
  • Mathematics — using mathematics to solve problems.
  • Science — using scientific rules and methods to solve problems.

Process

These are procedures that contribute to the more rapid acquisition of knowledge and skill across a variety of domains.
  • Critical Thinking — using logic and reasoning to identify the strengths and weaknesses of alternative solutions, conclusions, or approaches to problems.
  • Active Learning — understanding the implications of new information for both current and future problem-solving and decision-making.
  • Learning Strategies — selecting and using training/instructional methods and procedures appropriate for the situation when learning or teaching new things.
  • Monitoring — monitoring/assessing performance of yourself, other individuals, or organizations to make improvements or take corrective action.

Cross-Functional Skills

The paper also selected Programming from the list of cross-functional skills because of our prior knowledge about the models’ ability to code.
  • Programming — writing computer programs for various purposes.

Serious stuff- LLM Assistance And Research Statement

GPT-4 and ChatGPT were used for writing, coding, and formatting assistance in this project. Additional research is required to investigate the wider implications of LLM progress, including their potential to enhance or replace human labour, the effect on job quality, inequality impacts, skill development, and other outcomes. By attempting to comprehend the abilities and potential consequences of LLMs on the workforce, policymakers and stakeholders can make better decisions to navigate the complex AI landscape and its role in shaping the future of work.

Reference and useful links