Moderated by Aarti Bhaskaran, Global Head, Research & Insights, Snap Inc., the panel included Michael Nevski, Director, Global Insights, Visa; Suann Griffin, Former Senior Director, Insights, Serta Simmons Bedding; Jolanta Oliver, Director, Digital, Menu, Foundational Insights, McDonald’s; and Jennifer Ng, GM, Advertiser Insights, The Trade Desk.
The Synthetic Data Question
Aarti Bhaskaran: Such great points by all of you. I love the point of AI takes you there, but not all the way. Therein lies the balance. And I think we also forget that our stakeholders are humans. They need that human touch, that empathy, which is going to be missing for GenAI. But speaking of AI and research, I’m going to talk about a topic that’s now quite controversial or generating a lot of conversation, which is about synthetic panels. What are your thoughts or experiences with it?
Michael Nevski: For me, the synthetic panel needs to solve a few elements. Number one, efficiency. Number two, time to market. And number three, in the future, helping me to forecast to predict consumer behavior. While we’re looking at that, there’s a lot of talk right now about synthetic panels. But at the same time, I don’t see any robust or reliable solution yet.
Number one, I want to make sure that this synthetic panel can be benchmarked and measured against the actual sample where I can see that segmentation it provides or responses I can run quickly, and it would be similar to what I get in real life, from the real panel. Number two, if we build some kind of a productive predictability and again, actually measuring and understanding each aspect of your model. So GenAI has this kind of a black box, which is harder to adjust. They call it temperature, how the generative AI talks to you or responds. You can set it up at the same time, understanding each segment and adjusting each aspect or attributes of your study. It’s getting more challenging.
And number three, if we do the forecasting, how can I make sure that, say, if NPS scoring is swinging thirty basis points, how can we explain that? Can I look six months down the road? I relied on a synthetic panel, but they gave me this predictability score or propensity to purchase or something else in the future and how I can measure that and make sure that it’s going to be that versus the real human responses. I’d like to really understand how synthetic panels can be managed from our perspective and make sure that we can control all of those variables.
Suann Griffin: To give an analogy from 20 years ago, I was the insights manager at Shell, and everyone was petrified about moving our brand health tracker from phone interviews to online interviews. It’ll never work. We’re not going to have quality. This is before the days of bots and weeding out survey hackers. We tracked for a year and I hear Michael talking about forecasting. I think that we’re on that highway, we’re going 200 miles an hour and we have to do some tracking. We have to find money and in a time when budgets are tight and people are tight and teams don’t have bandwidth we need to find some time to be able to track and look at our historic approach and here’s the futures thinking approach and how do they track well together.
Do we have the human bandwidth and allocated on our teams to do those quality control checks, for this new AI. It’s not taking our jobs. It’s changing our jobs. We need to hold space and bandwidth to have that human thought process. People are hesitant about synthetic panels, but let’s not forget, we’re all dealing with the issues of “human panels,” and the representativeness of those panels in our research and those decisions. And so I think synthetic panels hold real promise.
It seems so early. It seems scary to make that switch. But people were really concerned about AI moderators as well. There’s a time and place for us to think about it. Do we need the human moderator, or is it the human element that is gauging what the AI moderator is doing. As Jolanta said, it takes us part of the way there, and then the human comes in and up-levels the moderator’s question to the respondents.
Jolanta Oliver: For me, it’s just coming back to the basics of what question do we want to get answered. Do we have the right business question in mind? Is a synthetic panel going to help get us there? And if it doesn’t, what else can we do to get to the heart of the question? It’s about prioritizing. What do you need to prioritize as a business to meet your strategy? Will synthetic panels help get you there? Or are they another complication that’ll cost money because you’ll have to test it over time?
I don’t have anything against it. I believe that if the business question is small enough, you could use AI to be able to answer it and get a little bit of the way there, similar to if you want to know about a company or know about a new area that may be growing in beverages or in food, you could maybe use a synthetic panel to see what are the all the types of consumers that talk about this kind of food and which one of those matches my segment and do a little research there. I don’t think there’s any harm in that, but at the end of the day, customer data that you own is treasure, and you’ve made a promise to your customer that you’re only going to use it in certain use cases. You can’t use it for everything. It’s like following along legally what you’re able to do with that data and not and just being very safe. Have your legal team be your best friends when you’re approaching using some of the customer data with a generative AI capability.
Bhaskaran: To me, it is also linked to the purpose of the research itself. Do you want to discover net new insight? Perhaps the synthetic panel is not the right tool to use. Do you want moderation at scale that are asking basic questions? AI is good enough. I think it also matters what purpose you’re using the research.
Check out the video for more of the Consumer Intelligence vs. Artificial Intelligence panel at the Road to TMRE 2024, plus a question-and-answer session afterwards. Click here for more of the content during Road to TMRE 2024.
