Data Quality Practices Key to Online Surveys
Your company’s data governance and data cleansing practices might also contribute to how usable the data is for its intended purpose. In this case, are the online surveys and their participants trustworthy? Many have raised concerns about the quality of online surveys. The growth of artificial intelligence and the proliferation of bots have only added to concerns. Bad data, notes Prolific, can lead to bad decisions by increasing the potential to come to false positive or false negative conclusions, for example.
Prolific, in its article, “How to improve the quality of data received from your online survey responses,” examines some key points in how to improve data quality from online surveys:
- Pick the platform with the strongest pool: The quality of your research platform will directly inform the quality of the data you receive from it. The marker researcher should evaluate participant verification, level of quality control, amount of data collected about participants, and active size of the pool.
- Validate your participants yourself: If you want high data quality in your online survey responses, make sure your participants are real people providing you with real data. You’ll have screening requirements that regulate who can participate in your survey. Ask them at the start of your survey to answer your screening requirements. This way, you can confirm that participants’ prescreening responses are accurate and up-to-date.
- Choose the right sample to answer your question: You won’t get worthwhile answers to all your questions from all available samples. Create a clear picture of your research aim, and build out a sense of what sample is likely to be able to give you the data to fulfill those aims. You need to ask the question: generalizability vs. nicheness — which one is more important to you? Generalizable samples give you the ability to generalize to the population at large, while niche samples are more useful if you want data about a very specific subset of the population.
- Design your study to eliminate bad actors: You need to make it as difficult as possible for bad actors to infiltrate your survey. This is primarily a technical and structural concern for when you’re building your surveys. Include CAPTCHA as your first step when it comes to maintaining data integrity. Include free-text responses, as these will reveal bad actors (and bots) through low-effort responses. Include open-ended and duplicate questions. If you suspect bot activity, check your data for random answering patterns or careless responding. Finally, implement effective attention checks to spot malfeasance in your database.
- Treat your participants well: There’s a human aspect to all this: If you want real people to give real answers to your survey questions, you need to treat them like real people. One of the best ways to do this is by paying them a fair rate for their time as well as being transparent about how the data will be used and how their privacy will be protected. Tell them about the benefits they can expect for taking the time to participate.
Focusing on the Survey Experience
All Things Insights looked further at surveys in “Improving the Customer Feedback Experience.” As the buyer journey becomes more complex, the customer satisfaction survey remains a foundational tool for the market researcher. With it, companies can measure consumer and brand sentiment among other factors. This isn’t just about bringing in new customers, but about retaining your older customers and making sure they are satisfied with their products or services. It’s a rather quick and convenient way to ask customers for their feedback, identify technical issues, and discover new opportunities. In addition, it helps maintain relevancy and a competitive edge while monitoring, and improving, progress and operations over time.
Looking forward to TMRE 2024? The conference, which will be held October 8 to 10, will feature the session, “How Google Detects and Combats Bad Actors in Large-Scale Online Surveys,” presented by Yerusha Nuh, Staff UX Engineer & Co-Lead of Research Execution at Google. Did you know that up to 40% of responses in market research panels can be classified as low quality? This not only costs researchers time and money, but also distorts the data and leads to unreliable insights. At Google, we survey millions of users every year to obtain insights that directly inform the development of our products. This talk will share findings on what Google considers to be problematic response signals beyond the traditional signals and some proposals for how we might detect them. Takeaways: pitfalls of traditional quality checks; techniques for detecting low-quality responses; and applications of AI-powered techniques. Register for TMRE 2024 here.
Improving Data Quality for Reliable Insights
Ensuring data quality in large-scale online surveys is crucial for generating reliable insights. We asked Gemini for some key ways to achieve this:
1. Targeting the Right Audience:
- Sample Selection: Use reputable sampling methods to ensure your respondents are representative of the target population. Avoid convenience samples that might skew your results.
- Screening Questions: Include screening questions at the beginning to identify and disqualify ineligible participants. This reduces irrelevant data and ensures responses come from your intended audience.
2. Encouraging Attentive Participation:
- Clear Instructions: Provide clear and concise instructions at the beginning of the survey to ensure respondents understand the purpose and expectations.
- Attention Checks: Embed subtle attention check questions throughout the survey. These can be factual statements or questions with illogical answer choices. Inattentive respondents will likely answer incorrectly, flagging their data for potential exclusion.
- Survey Length and Complexity: Keep the survey concise and avoid overly complex questions. Long and tedious surveys lead to fatigue and rushed, inaccurate responses.
3. Mitigating Bias and Dishonesty:
- Question Wording: Avoid leading questions or language that suggests a desired answer. Phrase questions neutrally to capture unbiased feedback.
- Social Desirability Bias: Minimize the influence of social desirability by avoiding questions where participants might feel pressured to give a “good” answer that doesn’t reflect reality.
- Verification Methods: Consider using CAPTCHAs or other verification methods to deter bots or automated responses.
4. Enhancing the User Experience:
- Mobile-Friendly Design: Ensure a seamless user experience across all devices, including smartphones and tablets. Poor design can lead to frustration and inaccurate responses due to technical issues.
- Logical Flow: Organize questions in a logical order that follows the natural flow of the topic. Jumping around can confuse respondents and lead to inconsistent answers.
- Visual Appeal: Use clear formatting, white space, and relevant images to make the survey visually engaging and user-friendly. An aesthetically pleasing survey encourages participation and reduces errors.
5. Post-Survey Data Cleaning:
- Incomplete Responses: Identify and potentially exclude incomplete surveys that lack key data points.
- Inconsistent Responses: Look for inconsistencies in responses from the same participant. This might indicate inattentiveness or deliberate dishonesty.
- Outlier Analysis: Identify and investigate data points that fall outside the expected range. These could be genuine outliers or errors.
Moving in the Right Data Quality Direction
Additional data quality practices could include pilot testing, with a small, representative sample to identify any issues with question clarity, survey flow, or technical problems. Implement data validation rules to ensure responses fall within the expected range. Be transparent about how data will be used and stored in the survey introduction. This builds trust and encourages honest participation.
By implementing these strategies, you can significantly improve the data quality of your large-scale online surveys. Remember, high-quality data is essential for drawing accurate conclusions and making informed decisions based on your research findings. Ultimately, data quality is a function of the participants, notes Prolific, and the study design. “Take care of your design and choose the right participants, and the right kind of data will come flooding in.”
Video courtesy of Enalyzer
