What is survey design? Survey design is a crucial research method, providing a structured approach to gathering data and insights. Whether you’re an expert or a beginner in UX design, learning how to make an effective survey is crucial for finding valuable user insights, and it can be challenging regardless of your experience. The essence of survey design is to create questions that gather specific information from a targeted group. When done right, it brings together psychology, language, and design to collect data on user preferences and behaviors. Surveys help validate hypotheses, understand user needs, measure satisfaction, and guide product development. The key is to approach survey design thoughtfully to ensure that the data collected are reliable, valid, and actionable.
In this article, we’re going to explore the art and science of survey design, including how to avoid common pitfalls and master the creation of your own survey with detailed guidance.
The main pitfalls of bad surveys
UX researchers frequently misuse and misunderstand surveys, blurring the line between qualitative and quantitative methods and potentially embodying the negative aspects of both. Let’s go over where a survey can fall astray.
Ease of use can be deceptive
Cognitive bias can cause survey findings to seem more valid than they actually are thanks to the simplicity of creating and distributing surveys and tallying results. Many researchers rely on surveys because they’re cheap and easy to distribute. But surveys are just one of the many tools you should use for research. It’s crucial to use various research methods, including surveys, to get the whole picture of what users think and do.
Risks of misguided decisions
Survey results often guide product teams in making critical decisions, but the inefficiency of surveys can lead to misguided directions. Again, surveys can’t be the only way you talk to users.
Difficulty in writing good surveys
Crafting effective survey questions is challenging. Surveys that are poorly designed can produce misleading data, and it’s difficult to correct these errors once they occur.
Bad surveys are hard to detect
Unlike other forms of user research methods, identifying a poorly constructed survey is tricky until the survey reveals contradictory information. Let’s put this into a real-world context: imagine you’ve created a survey to understand how users feel about a new feature in your prototyped mobile app. You asked a question like, “How satisfied are you with our new feature?” and offered a scale from 1 (not satisfied) to 10 (very satisfied). The results come back and look wonderful, as most respondents rate their satisfaction at 4 or 5. Then, after implementing the feature, you notice that user engagement with the app has dropped significantly, and there’s an increase in negative reviews mentioning the new feature. This contradictory information — high satisfaction scores vs. actual user behavior and feedback — flags that the survey might have been poorly designed. What could have been the issue? The survey questions might have been leading, or The scale didn’t capture the complexity of users’ feelings So, while the survey initially seemed fine, real-world user actions and comments revealed its flaws.
The illusion of quantitative certainty
Survey responses are easy to count, giving an illusion of certainty and objectivity, even if the data is inaccurate. Some people may make mistakes with their responses, leave questions blank, or not even read your questions correctly.
Misleading customer satisfaction metrics
We often use customer satisfaction surveys, but the data they generate may not accurately represent customer behavior or business success. For instance, the NPS has been a long-standing metric of customer satisfaction, but it’s been under fire for misclassifying active promoters and detractors, among other issues. NPS is important, but consider other factors when evaluating customer experience and business health. Secondary metrics that drive NPS, like churn rate, Customer Lifetime Value (CLV), and error rates in an app, give us a better understanding. By tracking these metrics, you can uncover areas to improve that businesses might overlook if they only concentrate on NPS.
Problems with survey design
One big mistake in survey design is the misalignment of question types and data needs. Unreliable data collection often occurs because designers present qualitative questions in a quantitative style. For instance, a survey might ask respondents to rate the usability of an app on a numerical scale. But, this approach might make complex experiences and feelings seem too simple. Instead, we could offer a text field where people can write about their experiences. This open-ended approach allows for more detailed feedback that a number rating alone can’t capture and gives us a clearer idea of what users think. Aligning the question format with the type of insight you seek is crucial for gathering meaningful and actionable data in surveys.
Lack of context in survey responses
Surveys frequently fail to offer the needed context for interpreting responses, particularly in the case of qualitative information. They can provide you numbers, but they can’t always provide the reasons behind them.
Not choosing the right research method
Using surveys is not always the answer. Before using them, evaluate if survey participants have the ability and willingness to provide truthful and accurate answers. If your survey responses consist mostly of qualitative data, observing human behavior is often more effective than relying on surveys to make decisions.
Using predictive or retrospective questions
Due to the unreliability of responses, it is best to avoid asking survey participants to predict future behavior or recall distant past events, like which competitors they’ve interacted with in the past year.
Not writing a survey research plan
UX designers often skip the research plan because of the rush of the product team. Taking the time to craft a research plan will pay back the effort. It will serve you as a guide to designing the survey questions well, as well as aligning your team on the goal. Here’s a detailed list of what to include in your research plan:
- Introduction to the research
- Overview
- Research problem
- Research objectives and goals
- Business impact of the research
- Expected outcome
- Research methodology
- Research design (qualitative, quantitative, mixed methods)
- Participant sampling
- Participant recruitment
- Budget and resources
- Incentive offered
- Data collection (tools used) and analysis
- Timeline
- References
- Previous research
- Documentation
- Citations
Don’t skip this step!
Transforming bad survey design: Common examples to learn from
We’ve covered the overarching issues with bad survey design. Now let’s take a closer look at transforming bad surveys. We’ll go through the most common issues you’ll see and teach you key survey design principles along the way. Here are some examples of common survey design mistakes:
Leading questions
Questions that nudge respondents toward a particular answer can skew results. To understand this, let’s analyze an example:
This is a leading question: “On a scale of 1 to 10, how satisfied were you with the user friendly and efficient shopping cart experience on our website?” (1 = extremely dissatisfied, 9 = extremely satisfied)
Why is it leading? Two reasons:
Assumptive language: The question assumes that the experience is “user friendly” and “efficient,” which may not be the case for all users
Bias toward positive response: By framing the shopping cart experience positively, it nudges the respondent toward agreeing that the experience was satisfactory
How can we improve this question? Let’s make one that’s not leading:
“On a scale of 1 to 9, how would you rate your experience using the shopping cart on our website?” (1 = extremely dissatisfied, 9 = extremely satisfied)
And these are the reasons this phrasing is better:
Neutral language: This question does not imply any particular quality of the shopping cart experience, allowing the respondent to provide an unbiased response
Open-ended: It invites a range of responses, giving the user the freedom to express their genuine feeling about the experience
Ambiguity
Vague or unclear questions can confuse respondents, leading to unreliable data. Here’s an example of an ambiguous vs. unambiguous question:
The ambiguous question goes like this: “How did you find the navigation on our website?”
Why is it ambiguous? Multiple interpretations: The phrase “how did you find” can be interpreted in different ways. It could mean “How did you come across our website?” or “What was your opinion on the ease of navigation on our website?”
Lack of specificity: The question does not specify which aspect of navigation it refers to, such as ease of finding products, the layout of menu items, search…
