The Risk Of Product Managers Relying on ChatGPT For Market Research
What it can and can't do well
If you’re a product manager, you don’t have to worry about ChatGPT replacing you. It’s not quite there yet. Still, it can serve as a useful tool as long as you accept some important caveats.
I’ve been toying around with it for a few different product management functions, and in this post, I’ll focus on market research.
To test this feature, I began by asking ChatGPT questions for which I already knew the answer, allowing me to validate if the answers it gave were true, complete, and inclusive of the necessary nuance one would need to draw the correct conclusions. I started with the following question:
Who are the major service providers for clearance and settlement services for banks, brokerages, and hedge funds that want to outsource their middle and back office operations?
ChatGPT listed all of the major players and gave accurate, if not sparse, information on each firm's strengths and weaknesses. I drilled down further by supplying a more detailed description of the type of customer I’m looking to target. ChatGPT answered with six possibilities, two of which did not fit because they did not directly sell outsourcing solutions even though they would be key players in an outsourcing implementation.
The real test came when I used my extensive knowledge of service providers for capital markets to purposely trip up ChatGPT. I asked which firms it recommended supported the processing of GCF repos. It suggested three firms but added a caveat, saying I should check and confirm because it wasn’t sure. At least it was being honest!
I then asked which of the firms supported partial pairoffs, a common function for middle and back office specialists. ChatGPT answered yes or no for each firm but lacked the nuance that a thorough analysis would reveal. Some firms support partial pairoffs for some products but not completely. Other firms might support it depending on your chosen configuration and the financial products you trade. Still others might support it but lack the integrations specific to a clients business needs.
These are important distinctions that are critical to uncover during the analysis phase. ChatGPT’s answer would give the wrong impression to someone who did not already possess deep expertise in this area of concern.
The risk of relying on ChatGPT
This demonstrates how you can get into trouble by outsourcing research to ChatGPT. Not only does it not provide the necessary nuance in its analysis, but what if you don’t know enough about the business to ask the right questions? You’ll walk away believing you understand the playing field when, in fact, you don’t even know the rules of the game.
Conclusion
Perhaps ChatGPT can give you a head start on your research. Even so, it cannot yet replace old-fashioned nose-to-the-ground research, talking to users, and interviewing users and subject matter experts experienced in your market. In short, use it as a tool to guide you where to dig in, not as a replacement for doing the hard work. Sorry, but there are no shortcuts.