Do you use ChatGPT in your work without a full understanding of what’s going on? The data analysis capabilities of the GenAI tool can be powerful, but one person working in market research has come unstuck presenting nonsensical analysis to clients. Thread follows. #techethics
Here’s the Reddit discussion with over 2300 replies at the time of posting “URGENT - my girlfriend used chatGPT for her work. Now her boss wants her to explain the calculations. I think the calculations were a hallucination. What to do?”
reddit.com/r/ChatGPT/s/LzxY4cqoYs
What happened? The Reddit poster shared a story about his girlfriend who works in market research, used ChatGPT to analyse survey data and create Excel spreadsheets. These were then used in a presentation for a client. The problem is, the outputs were fabricated or hallucinations.
Of course, the market researcher did not check or understand the outputs and could not explain them. It’s difficult to know where things went wrong, but it seems that qualitative data was processed as if it were numeric, in a way that anyone with even a basic knowledge of statistics would realise was wrong.
This serves as yet another reminder that tools like ChatGPT are not intelligent and they can’t replace foundational knowledge or the ability to direct a discussion. For users with expertise, they can be powerful, but everything needs to be carefully checked.
Was the market researcher here trying to reduce their workload, or to make out they had expertise that didn’t exist? Who can say. Did they even have permission to upload the collected data set to a GenAI tool?
Hallucinated data analysis is likely to become a much more common problem in the age of ChatGPT 5, which hides away so many of the underlying models and pipelines and removes user control. I’ve just come across the same problem myself.
I asked ChatGPT 5 to analyse the Reddit thread and responses. This, apparently, is what other Reddit users recommended the market researcher did to save their job. To me, it is obviously garbage, in the same way that student survey results that divide into convenient numbers are also often fabricated.
I did manage to get ChatGPT 5 to reason, develop a replicable methodology, and complete more accurate analysis of what the market researcher should do based on the top 500 comments (although only 498 were usable). This was more complex than necessary and is only lightly checked. And yes, Reddit users did encourage lying, but ChatGPT 5 has reasoned otherwise.
This point of this thread is not to share this specific example, but to serve as a reminder that foundational skills matter, as does an understanding of how to use GenAI tools and LLMs sensibly. The tools are not magic or intelligent. Educators need to teach students that GenAI use is much more than writing a prompt and assuming correctness. #academicintegrity and #professionalintegrity are both at stake here.