Predicting Product Success with AI-Powered Concept Testing
- Ben Harknett, CEO, Cambri
The vast majority of product launches within CPG continue to fail. This is something that has been and is still tolerated by many large CPG companies, who explain away this waste of money and natural and human resources by stating that innovation is hard and risky, that a high failure rate is the nature of innovation. But does it really need to be that way? By utilizing AI you can now combine quant, qual and in-market data to deliver more accurate and actionable insights.
“The benchmarks and KPIs that we have seen are…harsh, so they will filter out good concepts more than they will let weak concepts go through.”
Actionable Takeaways:
- Re-evaluate Traditional KPIs: Recognize that traditional KPIs like willingness to buy and uniqueness, when compared against benchmarks, may not be reliable predictors of in-market success.
- Leverage AI for Deeper Analysis: Explore AI-powered solutions that can analyze open-ended feedback and combine it with in-market data to provide more accurate predictions and actionable insights.
- Test Early and Iterate: Test concepts early in the innovation process and use data-driven insights to iterate and refine them, focusing on the drivers that have the biggest impact on market performance.
The Myth of Benchmarks: Why Traditional Concept Testing Falls Short
Ben Harknett, CEO of Cambri, took the stage at TMRE 2024 to challenge a long-held belief in the market research world: that benchmarks and KPIs are reliable predictors of product success. Through a compelling case study, he demonstrated the limitations of traditional concept testing and introduced an AI-powered alternative that promises to revolutionize the way we evaluate new ideas.
Traditional concept testing typically involves measuring KPIs like willingness to buy and uniqueness, comparing them against category benchmarks. However, this approach often leads to several problems:
- Defining “Strong” is Subjective: Simply testing above benchmark doesn’t guarantee in-market success.
- Conflicting Data: KPIs often contradict open-ended feedback, making it difficult to interpret results.
- Lack of Actionable Insights: It’s hard to determine which levers to pull to improve concepts based solely on KPIs.
- Time-Consuming Analysis: Manual analysis of verbatims and open ends is time-consuming and subjective.
The Case Study: Ready-to-Drink Disappointment
Ben presented a case study conducted with a large client in the alcoholic ready-to-drink (RTD) space. This client struggled with the disconnect between concept testing results and actual market performance. Products that tested well often flopped, while others that tested poorly surprisingly succeeded.
To investigate this issue, Cambri analyzed 30 RTD concepts, both from the client and their competitors, comparing their concept testing results with their in-market performance, measured by Weighted Rate of Sales (WROS), which factors in distribution. The products were divided into quintiles based on sales value, with the goal being to launch products into the top quintile.
The Shocking Truth: KPIs Don’t Match Reality
The results were striking. There was almost no correlation between how the concepts tested using traditional KPIs and how they actually performed in the market. Top-performing products often scored below benchmark on willingness to buy and uniqueness, while some poorly performing products tested quite well. The accuracy of the traditional KPI approach was a mere 33%, confirming that they were not good predictors of success.
Ben explained that the benchmarks were not necessarily wrong; they simply weren’t effective at predicting real-world outcomes. In fact, they were often “over aggressive,” filtering out potentially successful concepts. This meant that companies were likely missing out on valuable opportunities.
Why the Discrepancy? The Power of Open-Ended Feedback
The reason for this disconnect lies in the limitations of relying solely on quantitative KPIs. While KPIs provide a snapshot of consumer interest, they don’t capture the nuances and conditions expressed in open-ended feedback. For example, a consumer might say they’re willing to buy a product, but only under certain conditions, such as price or ingredients.
Traditional analysis struggles to incorporate this rich qualitative data effectively. This is where AI comes in.
The Solution: AI-Powered Concept Testing
Cambri uses AI and natural language processing (NLP) to analyze open-ended feedback at scale, identifying key themes, conditions, and drivers of consumer sentiment. This data is then combined with traditional KPIs and trained using in-market data (WROS) to create a predictive model.
This model generates a “Launch AI score” that accurately predicts whether a concept will perform well in the market. In the RTD case study, Cambri’s predictions were 90% accurate, a significant improvement over the traditional KPI approach.
A Real-World Example: Versatility Wins
Ben shared a specific example of a product that performed well in the market (third quintile) but tested below benchmark on willingness to buy. The AI correctly predicted its success, identifying “versatility of open-ended feedback” as a key driver. This meant that consumers expressed a wide range of positive reasons for liking the product, indicating broad appeal.
Beyond Prediction: Actionable Insights for Improvement
Cambri doesn’t just predict success; it also provides actionable insights on how to improve concepts. By analyzing the drivers behind the Launch AI score, the platform identifies areas of strength and weakness, allowing companies to focus their efforts on the most impactful changes. In the RTD example, Cambri identified “appearance” as a weakness, suggesting that the can design needed to be refined.
This data-driven approach removes subjectivity from the concept iteration process, allowing companies to make informed decisions about which concepts to pursue, which to refine, and which to abandon altogether.
The Future of Concept Testing: Data-Driven Decisions
The presentation at TMRE 2024 offered a compelling argument for moving beyond traditional concept testing methods. By leveraging AI and NLP to analyze the full spectrum of consumer feedback, platforms like Cambri are providing more accurate predictions and actionable insights, empowering companies to make data-driven decisions and increase their chances of product success. It’s not about replacing human intuition entirely, but about augmenting it with powerful technology to unlock the true potential of consumer feedback.