Watch the video on YouTube

ChatGPT is one of the latest projects from OpenAI, an artificial intelligence research laboratory based in San Francisco. It’s one of the fastest-growing apps of all time, having surpassed 1 million users just 5 days after its launch on November 30, 2022.

Essentially, ChatGPT is a chatbot program that is designed to be able to have natural, human-like conversations with people. It uses machine learning techniques to analyze input and generate appropriate responses.

In fact, the last two sentences you read were generated by ChatGPT with no edits. I just asked it to explain ChatGPT in simple terms.

While we’re far from a general-purpose AI, ChatGPT is surprisingly skilled at drafting songs and essays, analyzing and summarizing data, and writing and explaining computer code.

Anyone can access ChatGPT by creating a free account at chat.openai.com. Beware: it’s not perfect. Responses can be overly verbose, redundant, and even contradictory. You need to fact-check. Use ChatGPT as a tool to assist you, but don’t trust it blindly.

Despite its flaws, I propose that this style of conversational AI is going to replace our current concept of Googling in the near future. No more “x + reddit” Google queries for those fed up with the current flavor of PageRank.

In the meantime, I believe ChatGPT can be a powerful tool to assist those involved with product management, design, and development, with limitations. Today, I want to demonstrate some examples.

ChatGPT can be used to generate a list of product or feature ideas.

Example prompts:

ChatGPT is conversational, so you can ask it to elaborate on previous answers in the same chat thread. I asked for more ideas:

I asked how it would go about designing one of the features it suggested:

What I got back was helpful, but generic. So I asked it to outline some specific considerations users might have for this feature:

Quite insightful. Next, I asked for pain points:

Finally, I asked for a problem statement:

Now that we have generated some ideas and elaborated on one of the features, it’s time to understand our users at a deeper level.

I asked ChatGPT to generate personas for the example feature:

These are all useful, and it’s nice that ChatGPT tied the personas back to the pain points outlined earlier. You could argue that these personas could be combined, as the roles are all related to production and logistics with similar problems and needs.

What I think is missing in these personas is roles on the business side—sales, customer support, and management. Each of those personas may have an interest in inventory analytics for very different reasons.

I asked ChatGPT for suggestions:

On the surface, these personas are decent. However, if you analyze them, they are not exactly what I would expect from a strong product manager. Let’s unpack them one-by-one:

It is possible the needs of certain stakeholders are not going to satisfied by a minimum viable product (MVP), but it is important to consider these personas at this development stage.

ChatGPT’s responses in this section demonstrate the limitations of the current language model. It displays a remarkable general awareness of topics, but it does not replace the industry knowledge and level of deep inquiry a good product manager would demonstrate.

First, I asked ChatGPT to describe the Slack app:

ChatGPT seems to have forgotten that this is a plugin for Slack (it never explicitly acknowledged this in any of the responses), and it is essentially rewriting a feature list into paragraph form.

I’m going to provide some more context and ask it to be more focused in its output:

This reads more like a press release than ad or website copy. And I think the app needs a name at this point:

This is not terrible copywriting, considering how little effort I’ve put into this. It doesn’t seem to know about the specific character limits for facebook ads, so I’ll give it some more guidance:

Perhaps a bit redundant on the inventory management aspect, so I’ll just ask it to generate some alternate headlines and descriptions I can A/B Test:

It seems to have forgotten about the 30-character limit on descriptions and the fact that I asked for only 5 variations, but who can really complain about more? I think you get the idea that it’s a starting point that might need some retries or edits.

Let’s make a landing page focused on a specific user:

I don’t think ChatGPT understands the specifics landing page formatting, as this is more like video or audio ad copy. I asked it to create a different page, with more specifics about the structure:

We can certainly nitpick, but this is a decent first draft to mark up and hand over to a copywriter.

For SaaS companies, typically, a key objective is to release a minimum viable product (MVP) to your target users as early as possible. ChatGPT can help you reduce time to market in a couple ways.

The first is with helping you write code. Thanks to OpenAI Codex, the same engine behind GitHub CoPilot, ChatGPT can generate code snippets based on a natural language description.

I got excited and asked it to write a whole script for me, and it pushed back, but it provided a helpful response about how to design my script:

I got more specific with my query. This time, it gave me some code:

I didn’t test the script, but I appreciated that it added comments and an explanation. Generally, it seems to be about right. It even implemented the dedupe logic in a straightforward manner. One small thing: it would have been nice to explain the setup of the Inventory spreadsheet.

Now that we have a script for our MVP, let’s populate the inventory spreadsheet with dummy data:

I was expecting a list that I could just copy into a spreadsheet, but the auto-generating code could be useful. My presumption is that I originally asked for code, so ChatGPT assumes I still want code.

Without having run the code, it looks generally right, but I believe it will crash in the for loop because there are fewer than 50 items in the toys list. Maybe it should just loop over the length of the list instead. Also, it will populate the list with a quantity, but it’s just going to progressively be 10, 11, 12, etc. instead of something more realistic. A random value in each row would be better.

I asked it to just give me a list I could copy:

Just like the code, there are only 42 items in the list. But the data is usable.

The code and spreadsheet we just generated are sufficient to build our MVP and start showing the product to users.

With our MVP in the hands of users, we’ll want to start tracking user behavior and getting ideas for how we can improve the product.

I asked ChatGPT what user signals I should look for:

This is quite generic, but it’s a fair combination of metrics and more qualitative input. Let’s ask ChatGPT to design a user survey:

Not bad. I’ll give it a few more inputs to generate a tighter survey:

Pretty good! I’d reorder the questions, but these are usable. Just keep in mind that our MVP doesn’t have all the features listed (I didn’t explicitly specify this to ChatGPT so it’s fair).

I asked ChatGPT to generate some dummy responses to the survey, mainly to test ChatGPT’s contextual awareness, and it did pretty well:

Finally, I asked ChatGPT for metrics:

The reorder rate suggestion is a little weird, since it was pitched as an automatic feature, and it’s also not in the MVP. To be fair, we never explicitly specified that this would not be in the MVP, but this is an example that ChatGPT’s contextual awareness is not perfect.

The other two suggestions are fair, but I think retention is more important than any of the metrics presented here. Perhaps the metrics could be weekday daily retention, weekday DAUs, and NPS.

This article took me a few days to write, and I ended up running all the queries again because I noticed the output was getting more insightful as the days went on. Indeed, OpenAI released the January 9 model as I was writing this post. I ended up rewriting the article because my opinion of ChatGPT became more favorable after switching to the new release.

It’s interesting to me that, subjectively, a language model can improve so quickly. The December 15 release was giving me somewhat insightful but generic responses. The January 9 model was providing deep insights about problems faced in the manufacturing sector. It also became more contextually aware. Disclaimer: n=1.

I still wouldn’t fully rely on ChatGPT’s output. The language model has been known to be confidently wrong about many topics, and it might give you blind spots, as demonstrated in this article. But considering how fast the results seem to be improving, I can only imagine how much more useful tools like this will be a year or two from now.

With that said, I wouldn’t be worried about ChatGPT wiping out product management and design roles any time soon. To be successful in product, you still have to talk to users and understand their pain firsthand. Creative roles require a combination of empathy and technical skill, along with a willingness to manage significant complexity.

Ultimately, ChatGPT is another tool in the toolbox, allowing us to brainstorm and create better designs faster.