Skip to content

Data Quality Challenges Raise Questions as Community Eyes Future

Download the report now.

Overall, things seem to be looking up for those in the data science community. For example, respondents feel strongly that analytics and data science are becoming more integrated into both corporate and operational decision making. While recessionary winds keep blowing, and many feel that doing more with less is the status quo, the report indicates that the analytics and data science discipline continues to feel positive about its growth prospects. Over 60% of the community feels that influence has grown post-pandemic. Nearly half of the community is focused on growth.

Questioning Data Quality

Still, there are some challenges that survey participants indicated. An interesting challenge to currently explore is that of “data quality.” When asked, “Which are your organization’s top challenges in using data to inform decision-making?,” the majority of respondents (55.9%) said that data quality was the top challenge. To be sure, there were other challenges also selected in the survey: 39.4% said that a continuing challenge was the budgets/resources for data science. Meanwhile, talent/hiring the right skill set was selected by 33.1% of respondents.

Yet, back to data quality as the top challenge. Just what is driving that response? The rapid development of technology in the field could be one possible answer. Whether or not you think that the function is in part responsible for data quality, with the onset of Generative AI and LLMs, data quality must continue to be an area of focus based on the continued amplification of possible usable and useful data sets. This could also in part answer the question of why securing talent is also such a challenge, as the rise in technology presents its own set of issues in terms of training, upskilling and recruitment.

For her part, June Dershewitz, Board Member, Digital Analytics Association, challenges the community to change its mindset regarding data quality. Dershewitz asserts, “I think for organizations to say their top challenge is data quality is a scapegoat. It’s always there. I’m actively involved in conversations and initiatives aimed at improving data quality. We can do better. It’s hard to solve because it has to be solved collaboratively across the business with different parts of the organization that have a hand in creating the problem.”

Dershewitz further notes, “Few respondents said that there was not enough influence on the business and yet they only said that they were somewhat integrated into business decision making. That’s why I think data quality is kind of the excuse that’s masking the true problem, because if you had that influence on the business, you might be able to more strongly advocate for putting resources where they need to be to improve data quality.”

Perhaps an additional cause of data quality issues is the speed of technology in the current reality. For Neil Hoyne, Fellow, Wharton, the strategy playbook is rapidly changing and companies must be flexible and agile. “The persistent focus on data quality risks slowing things down. This prerequisite is an almost never-ending obstacle to any decision-making. Instead of striving for perfection on their data, organizations should focus on delivering incremental progress on the current programs. In other words, can leaders have the discipline to prioritize data-driven growth over perfection?” says Hoyne.

That’s not to say that data quality issues are irrelevant or nonexistent in the community. This also directly relates to the eventual generation of insights.

Hoyne adds, “If you don’t have good data, you won’t get anywhere. You won’t be able to influence or understand anything because you won’t even get to the analysis stage. Data quality will continue to be a top concern until companies can address it. But again, I believe some companies are so obsessed with data quality and afraid of imperfect analysis that they’re slowing down their progress. We need to ask ourselves if we’re telling the right story. Data is the language that tells us what’s happening in the marketplace. It’s time for data and analytics to step up and fulfill their role.”

The Volume of Data

The sheer volume of data being generated these days seems to be a looming concern among data science professionals.

“Data quality and the ability to translate data into insights into the business is a top concern,” agrees Anu Sundaram, Vice President, Business Analytics, Rue Gilt Groupe. “Top challenges include using data to inform decision making and data quality. I always hear data quality being the issue. We just can’t get it. There’s so much more data now, and the quality needs to catch up with the volume. The inability to translate data to the business is also very close to lack of data in the survey.”

Sundaram raises an interesting point, that the heightened volume of data is causing a logjam of sorts to being better able to understand the data.

“Respondents are doing what they can do with what they’ve got and while they are not doing poorly, have concerns regarding data quality issues,” adds Sundaram. “They need better processes, better resources, better ways to synthesize those data elements. Only then will you get value out of it.”

Chuck Martin, Editorial Director, Informa Tech, also sees the volume of data playing a role here and how it impacts quality. Human employees, it seems, have not been able to keep pace with the development of new technology.

Martin says, “Many organizations in the survey selected data quality as a challenge. At the Gartner data analytics conference, everything was about we have too much data. We can’t figure out which data we should get rid of. We can’t figure out how to get the data to work together with the other data. It’s just a mismatch they’ve created over the last few years.”

He continues, “The capability has risen to capture a lot of data and now they’re trying to figure out what we do with all this data because it’s not necessarily all useful. It’s a fire hose coming in. That’s a problem because they spent all these years figuring out how to capture all the data. Now they have to figure out which data they actually do not need.”

Still, the amount of data is in some ways a “good” problem to have for the data science community, which has shown itself to be resourceful and resilient. Getting a handle on all this data, to control what’s coming in, could in turn better support operations in the future.

“The data gets you there or doesn’t and there’s not much you can do with it if you can’t control what’s coming in,” says Matthew Mayo, Editor-in-Chief, KDnuggets. “It’s an ongoing struggle every day, no matter what the size of what you’re working with, even with ChatGPT and these huge models. Data quality could always be better. The data you need is probably out there somewhere. And if you can’t get it that could be a question of resources.”

The survey showed that data quality or lack of data or inability to translate data to the business was a common cause of concern. Mayo notes, “That’s interesting, that inability to translate. If it’s your organization’s data then not being able to make it work for you, that seems like a different kind of problem. This might be a lack of data fluency and the data scientists having an issue integrating their data into the business. Stakeholders want results. It’s not a matter of the data itself. It’s a matter of the process and the people working on the processes with the data.”

Moving Forward

Clearly, there are some concerns about data quality among the community, and there is room for improvement in that regard. However, that could also prove to be an opportunity for the skilled data specialist or data science team.

As Michael Bagalman, Vice President, Business Intelligence and Data Science for STARZ, puts it, “The exact nature of these data quality challenges is unclear, but the implication is that there is a significant opportunity for improvement. Despite budget constraints, we can anticipate an escalated emphasis on data engineering and data hygiene in the future. Given that these elements form the bedrock of data management, it aligns perfectly with the prevalent theme of efficiency, the need to address and rectify ongoing issues.”

Bagalman adds, “Data quality inevitably assumes paramount importance, as it forms the foundation on which all subsequent data-related tasks are built. After investing considerable resources, it is imperative to ensure not only that your data is of high quality but also that it is structured correctly to support your specific requirements.”

Just who is responsible for the data quality at an organization might also be a potential question that can be raised within the analytics community.

For Michelle Ballen-Griffin, Head of Data Analytics, Future, “Data quality is everybody’s problem. The whole organization needs to be involved in data quality. It’s not just the data team’s role. You have to put controls in place and have data governance. Educate people on their decisions and how it influences quality.”

With the rapid advancement and developments of technology, data quality is an issue that probably won’t be going away anytime soon. That might challenge the industry to improve its training and career development as well.

KDNuggets’ Mayo notes, “Some survey respondents did say they don’t know what to do with the data. That stands out, whether it’s good or bad. It makes sense but it seems like something that you’d want to take care of if you want to better influence and integrate with the business. There’s a lack of machine learning and predictive modeling preferences in the survey that is in line with what I thought. But with technology advancing, it’s a prime opportunity to learn about and incorporate new tools into the business.”

The All Things Insights analytics and data science community that participated in the H1 2023 Analytics & Data Science Spend & Trends Report has a solid core of top-level leaders along with a fair share of front-lines operators. The community is spread across several different industries with a fair share of large organizations represented along with some smaller companies.

Download the report now.

Video courtesy of Eye on Tech


  • Matt Kramer

    Matthew Kramer is the Digital Editor for All Things Insights & All Things Innovation. He has over 20 years of experience working in publishing and media companies, on a variety of business-to-business publications, websites and trade shows.

    View all posts

Related Content

Building Empathy to Improve Insights

While customer centricity is a key attribute in today’s insights and data-driven corporate culture, success must also be built on a concept of empathy…

Driving Analytics & Insights Action

At this year’s TMRE @ Home, Shivani Shah, National Category & Shopper Insights Manager, H&WB, Church & Dwight Co., Inc., presented the session, “Data…


More Related Content

Happy family of four and dog walking on the beach by the ocean.
data science

Building Empathy to Improve Insights

While customer centricity is a key attribute in today’s insights and data-driven corporate culture, success must also be built on a concept of empathy…

Many colored arrows pointing to the center of the graphic.
actionable insights

Driving Analytics & Insights Action

At this year’s TMRE @ Home, Shivani Shah, National Category & Shopper Insights Manager, H&WB, Church & Dwight Co., Inc., presented the session, “Data…

Vortex-like laser images in space like setting.
data science

Finding the Right AI Research Opportunities

As part of the AI & Tech Summit portion of this year’s TMRE @ Home virtual gathering, David Iudica, SVP, Research and Insights: Retail and Wealth, Cit…

A big magnifying glass.
consumer behavior

Searching for Cultural Anthropology Insights

Consumers and brands are influenced by culture. Our values and identities, among a variety of societal and social factors, are tied to cultural contex…

Hands shaping a clay pot on the potter wheel.
insights strategy

Positioning the Market Research Mission

Just how should the enterprise and the team go about determining its market research mission? This specific market research mission is not necessarily…

soccer player on his knees upset that he lost or missed goal.

How Insights Can Benefit from Failure

No one actively looks to launch an unsuccessful product or venture. Yet failing can have its upside. There is a philosophy of thought in the business…

Book with pages flying open.
insights talent

Learning the Soft Skills of Market Research

Skills can be developed in a variety of ways in the insights workplace to create an effective, versatile employee with a diverse arsenal of competenci…

stressed out worker in front of laptop
insights talent

Stay Competitive Through Upskilling

Upskilling and reskilling—training workers so they can fill more advanced roles in an organization—is becoming vital in today’s business environme…

Gold bars
media insights

Determining the Value of Business Content

Michael Bagalman, Vice President, Business Intelligence and Data Science, STARZ, gave a presentation at the Media Insights & Engagement Conference, ca…

data science

Defining Governance to Deliver Data Benefits

Big data is here, it’s a growing trend and it’s not going away anytime soon. The rise of artificial intelligence and other technological developments…

data science

Turning Data into Market Intelligence

When surveying a new market opportunity, the company’s market intelligence efforts come into play. This valuable data helps determine market segmentat…

data science

Diving Deeper into Insights ROI

The third day of TMRE@Home, May 2, will demonstrate best practices involved in developing and positioning insights ROI. Fusing AI, data and analytics…

data science

Blending Analytics & Insights

The second day of TMRE@Home, May 1, will demonstrate best practices involved in marrying analytics with insights. Insights from analytics can be an ev…

machine learning

Plugging into the Future With AI

TMRE@Home is going to be held virtually this year from April 30 to May 2, 2024, and it will feature several educational tracks for insights and market…

actionable insights

Building the Analytics & Insights Ecosystem

The recent TMRE Continued virtual event hosted a session, “Blending Analytics & Insights to Drive Business Outcomes.” Moderated by All Things Insights…

insights and analytics

New Frontiers for Capturing Consumer Sentiment

Kajoli Tankha, Senior Director, Consumer Marketing Insights, Microsoft, and David Evans, Senior Research Manager, Microsoft, held the session, “Captur…

insights talent

Evolving Your In-House Team

During TMRE Continued, All Things Insights’ Seth Adler had a chance to hold a session with Shilpa Khanna, Assoc. Dir., Transformational Growth Insight…

insights strategy

Increasing Insights Influence

Thomas Walker, Senior Director, Global Brand and Customer Insights at eBay, and Christopher Frank, Author, Co-Faculty Director at Columbia University,…

consumer insights

Operationalizing a Hybrid Approach to AI

In “Insights + AI: Operationalizing A Hybrid Approach,” Christina Speck, Vice President, Corporate Strategy & Commercial Product, Blue Cross & Blue Sh…

data science

Finding the Right Machine Learning Formula

In “Machine Learning: Optimizing Ad Campaigns,” Michael Bagalman, Vice President, Business Intelligence and Data Science, STARZ, explored machine lear…

consumer insights

The Evolution of Building an Insights Team

Josh Goldfinch, Director Customer Insights, Verizon, participated in a panel at TMRE focused on “Building an Insights Function from the Ground Up.” Al…

insights culture

Navigating to Insights ROI

All Things Insights’ Seth Adler had a chance to sit down and get cooking with Jordan Cusner, Head of Guest Insights, Popeyes Louisiana Kitchen at TMRE…

consumer behavior

Infusing the Behavioral Insights Mindset

During the “Behavioral Insights” session of the Road to TMRE Gathering, Seth Adler explored “Activating Behavioral Insights: Team & Consumer Themes” w…

insights leadership

The Strategy Behind Insights Influence

Josh Goldfinch, Director Consumer Insights, Verizon, presented a panel discussion on leadership and influence at TMRE @ Home. All Things Insights’ Set…

Data Science & Analytics Strategy
data analytics

Reinventing the Data Relationship

The reinvention of the relationship between enterprise data science and business stakeholders is upon us. The focus is now on what the data can do for…

actionable insights

Diving Deeper into Measuring Insights ROI

If you can measure it, you can manage it. Insights leaders came together on this panel discussion on “Measuring Insights ROI” to share their respectiv…

data analytics

Demystifying Insights ROI

Key challenges come into translating business insights into ROI. This includes data quality, research bias, no direct translation of insights into bus…

actionable insights

Establishing Insights ROI

In a recent ATI Gathering…Insights ROI, the community looked at the complex issue of insights return on investment from several different angles. Co…

data analytics

Unifying Data Analytics and Insights

To be human or not to be human? That is a relevant question in today’s technology-driven society, and one we posed when thinking about unifying data a…

data driven culture

Gaining Insights Leadership

The role of today’s Insights professional is evolving as Insights teams are increasingly pressured to connect data sources, surface consumer stories a…

television audience measurement

Measuring OTA Programming

Television audience measurement is on the precipice of innovation due to the opportunity to leverage data from set top boxes, smart TV’s, and other so…

strategic insights

Building Strategic Insights

The world of data is evolving and ever-changing. A data insights strategy is built in part on the potential data that one hopes to accumulate, the ins…

data translation

Translating Data Into Action

“Data is the new oil,” and it seems like everyone wants to get a piece. But what is it that most major companies are missing when it comes to harnessi…

data analytics

Interpreting Data

Data isn’t just a set of numbers in a spreadsheet that sits there collecting digital dust. Michael Nevksi, the Director of Global Insights at Visa, no…

data analytics

Data Fluency

Data fluency and how it applies to an organization is essential for a business to thrive – according to Bob Bress, Vice President and Head of Data S…

data analytics

Digestible Insights

Business founders and leaders are proud of their accomplishments and with good reason. They have led their operations, often for decades and through s…

insights strategy

Timely Insights

“May you live in interesting times.” We are all familiar with this curse (or blessing, depending on how you see it). Many of us even know it as Chines…

insights strategy

Insights Enablers

Over the past two and a half years, there “have been more DIY projects” thus “Insights teams are moving more to becoming consultants and enablers fro…

data science

Pandemic Analytics Insights

It is reported that after the pandemic that 50% more customers and teams will use Business Intelligence tools. “Adapting, moving online, and having a…

actionable insights

Data, Analytics, Insights & Action

If you watch June Dershewitz when she’s not looking, you might catch sight of her cape. She is a superhero in the world of data. Her goal is to make t…


One Way To Insights ROI

“It’s not only a ‘feel good’ thing anymore. It is actually something that is reflected in profits.” DEI is a business issue. Once nice to have, the co…

data analytics

Insights On Outpacing Further Disruption

The axiom, ‘May you live in interesting times,’ is cited as both an expression as well as a curse. Its origin is foggy. For the past couple of decades…

user experience

UX Insights

Shilpi Sinha is not only speaking at TMRE but joining the pre-event Innovating Insights Workshop as a table co-host. Both the session on the agenda an…

data science

Data Culture

Global society and business continues to shift from decisions informing data to data informing decisions. That is to say, for most of human history, a…

data science

Insights Resilience

Way back at the beginning of our modern almost apocalypse, there was one term used across the globe and up and down the organization- resilience. Faci…

insights leadership

New Work Order: The Future Of Work Is Here

2022 still has some work to do in figuring itself out. We’ve clearly not yet definitively evolved to what’s next. That said, some patterns have emerge…

insights talent

Internal Insights

Think back to the office of February, 2020. In many ways it felt like the office in February, 2010 and February, 2000. For the most part, every single…

data measurement

Measurement, Currency and Attribution

Keeping pace with innovation is always a challenge. The Media industry not only has to keep pace with that innovation, many players have offerings tha…

insights strategy

Mental Fitness In Business Culture

In the first century, it was the Christian Church which accounted for a substantive portion of societal change. In the fifth century, it was Greek Dem…


Insights From Seizing The Moment

Remember the young students at Welton Academy of “Dead Poets Society,” huddled around the photos of their predecessors who were no longer living. But…


The Missing Link Between Data And Business

2022 has proven to be a year of change. While lingering reminders persist, the past two years are falling further and further back in the rear view mi…