This deep dive guest post written by Senior Global Product Manager, Jin Hu, delves into the opportunities that lie ahead for product managers with the arrival of ChatGPT and other advances in AI.

---

Hi there! As a product person in back office operations, I always try to find ways to improve efficiency, even if it means eventually eliminating some jobs (please don’t hate me, this is often a consequence of doing my job well). Lately, I’ve been really interested in exploring how GPT can help streamline my work process and improve the productivity of the products I’m responsible for.

A few days ago I came across a paper titled “GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models” by authors from OpenAI, OpenResearch, and the University of Pennsylvania. It’s been getting a lot of buzz, with several YouTubers discussing its findings. I highly recommend reading the paper or checking out some of the videos, I left some links at the end of this article.

I initiated a chat with GPT out of pure curiosity regarding a table mentioned in the paper (Screenshot below). This table describes the regression of occupation-level, human-annotated exposure to GPTs on skill importance for each skill in the O*NET Basic skills category, plus the programming skill.

To explain this to a 5-year-old, the higher the number in the table the more the skill is likely to be augmented/exposed by different levels of LLM involvement (𝛼, 𝛽, 𝜁. search the paper for more detail). Exposed here means “a measure of whether access to an LLM or LLM-powered system would reduce the time required for a human to perform a specific DWA or complete a task by at least 50 percent.”

The conclusion made in the paper on this table is the same as you can see here.

The higher the number, the augmentation is stronger, and the negative number means the LLM could not be of assistance to augment the specific skill.

Now we have set the stage. Let’s continue to follow my journey.

I wanted to know how product managers’ day-to-day tasks mapped with the O*NET Basic Skills and find out how many of them required “science and critical thinking”. Here’s a table that GPT generated based on my requests:

Table 1. A list of generic product manager tasks mapped with O*NET Basic Skills also other skills needed identified by GPT. Each row is the skills required to complete the respective task successfully.

Please note, this is a list of tasks for an individual contributing product manager, not for people managers. Also, I am well aware this isn’t a comprehensive list of tasks nor the O*NET skills matched perfectly. I still decided to publish it as-is, given this isn’t the main point to make here and it’s good enough.

There were a few “errors” from GPT that I had to correct during the chat; otherwise, the result was very impressive. Here are three types of errors made:

• No clear explanation on why a task can/can’t be assisted by GPT, e.g., GPT initially said it couldn’t help with user research. I argued otherwise. I broke down the research process into more specific tasks and pointed out which tasks could use GPT for productivity.
• GPT picked up some more niche requirements for product managers and not necessarily applicable to all product managers, e.g., GPT initially suggested product managers are responsible for pricing.
• GPT failed to pick up some generic product manager’s day-to-day tasks, e.g., stakeholder management.

From time to time, there are similar errors in one answer from GPT. When I asked it to correct the error, it normally also picked up the other ones and provided a better version of output.

Based on the statement from the paper and combined the table from GPT, I conclude more than half of the PM tasks won’t be exposed to GPT4 (current algorithm). Saying this, I wanted to know in more detail, exactly which are the tasks the GPT can or can not help with. After a while, our chat grew closer to what I was looking for. Here’s the result, a table listing generic product managers’ day-to-day tasks, whether GPT can be of assistance, and how.

Table 2. Continue using the same set of the generic product management tasks, based on the description of each task to see if GPT recognise this is something GPT can be of assistance, and in what way.

As you can see here, there is no single definitive “Yes” in the table, only some “Maybes.” You might also find that the “Maybe” in this table isn’t exactly the same tasks as the tasks that require Critical thinking and science as skills. I think two reasons, first is that even a big part of a product manager task requires Critical thinking and scientific research skills, but they still can be broken down into smaller parts that could be assisted by GPT.

Second, the context between Table 1 and Table 2 is different. Table 1 is about mapping between the O*NET Basic Skills and the product management tasks based on a statement from the paper. Whereas there is no pre-defined list or standard for GPT to reference on when we generated Table 2, all information is gathered based on what’s available for the current GPT version.

Until next time …

In conclusion, our conversation was a fascinating exploration of the essential skills of product managers and how GPT could assist with them. Here is a list of tasks you can try getting help from GPT. Some of them require GPT to have access to more context, eg. the detail and background of the program you’re working on, etc. Also, some of the work that could be done by GPT should in theory be further improved by apps built on top of AGI with specific instructions.

1. Conduct market research: GPT can assist with this task by analyzing online forums, social media, and other sources of data to identify trends and patterns in consumer behaviour.
2. Identify user needs and pain points: GPT can assist with this task by analyzing user feedback and identifying common themes and issues that need to be addressed.
3. Create user stories: GPT can assist with this task by generating user stories based on user feedback and requirements. The premise is that GPT has access to all the context of the company and the program you are part of.
4. Develop product positioning and messaging: GPT can assist with this task by analyzing market trends and consumer behaviour to develop effective product positioning and messaging.
5. Create a Go-To-Market strategy: GPT can assist with this task by analyzing market trends and consumer behaviour to develop effective Go-To-Market strategies.
6. Product marketing: GPT can assist with this task by generating marketing copy and content based on market trends and consumer behaviour.

While GPT could provide some assistance in certain areas, human input and decision-making were still crucial in ensuring the success of product management. It was clear that while AI has the potential to revolutionize the field of product management, it could not yet fully replace the skills and expertise of a human product manager.

Appendix- O*NET Basic Skills

Content

These are background structures needed to work with and acquire more specific skills in a variety of different domains.

• Reading Comprehension — understanding written sentences and paragraphs in work-related documents.
• Active Listening — giving full attention to what other people are saying, taking time to understand the points being made, asking questions as appropriate, and not interrupting at inappropriate times.
• Writing — communicating effectively in writing as appropriate for the needs of the audience.
• Speaking — talking to others to convey information effectively.
• Mathematics — using mathematics to solve problems.
• Science — using scientific rules and methods to solve problems.

Process

These are procedures that contribute to the more rapid acquisition of knowledge and skill across a variety of domains.

• Critical Thinking — using logic and reasoning to identify the strengths and weaknesses of alternative solutions, conclusions, or approaches to problems.
• Active Learning — understanding the implications of new information for both current and future problem-solving and decision-making.
• Learning Strategies — selecting and using training/instructional methods and procedures appropriate for the situation when learning or teaching new things.
• Monitoring — monitoring/assessing performance of yourself, other individuals, or organizations to make improvements or take corrective action.

Cross-Functional Skills

The paper also selected Programming from the list of cross-functional skills because of our prior knowledge about the models’ ability to code.

• Programming — writing computer programs for various purposes.

Serious stuff- LLM Assistance And Research Statement

GPT-4 and ChatGPT were used for writing, coding, and formatting assistance in this project.

Additional research is required to investigate the wider implications of LLM progress, including their potential to enhance or replace human labour, the effect on job quality, inequality impacts, skill development, and other outcomes. By attempting to comprehend the abilities and potential consequences of LLMs on the workforce, policymakers and stakeholders can make better decisions to navigate the complex AI landscape and its role in shaping the future of work.

Reference and useful links

• GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
• O*NET Basic skills category
• GPT 4 Can Improve Itself — (ft. Reflexion, HuggingGPT, Bard Upgrade and much more)