Creating good surveys and analyzing the data obtained through surveys is a challenging task. Answers can differ widely depending on the type of survey and question used to gather that data. For example, the choice of questions selected in a survey could be priming the respondents to answer in short, one-word answers. Consider the following examples.
A. Are you satisfied with the service?
B. How can we improve our app?
Questions like A. tend to trigger minimal answers such as “yes” or ”no”. On the other hand, questions could also be formulated in a way to allow the respondents to be more expressive and freer in their feedback. Questions like B. tend to be answered with a longer and more explanatory response.
If you want to learn about your customers’ opinion of a certain product, a simple “yes” or “no” answer will probably not be too insightful. In order to get the most insights out of your survey data, you should think about the following points before you start gathering feedback:
- What type of data do you wish to obtain through the survey?
- What type of survey and questions will trigger the type of data you are interested in?
- How do you plan to analyze the data obtained?
- How do you want to summarize/visualize/use the results from the data analysis?
In this article, we will address the questions above and provide the reader with some best practices when it comes to choosing the right survey for your particular use case.
Based on our experience with a large variety of datasets at Gavagai, we have identified 3 types of surveys:
- Surveys with definite answers
- Surveys with leading questions
- Surveys with open-ended questions
All three survey types have their advantages and their results can provide valuable insights, if they are analyzed with the adequate method. Let’s discuss them one by one.
Surveys with definite answers are of the multiple-choice type, i.e. the survey takers are given a selection of answers from which they can choose. The number of definite answers can vary depending on the survey’s purpose. In rating scale answer fields, the number of choices given are usually higher than for example in multiple choice fields, where the respondents might just choose between “yes” and “no”. In some cases, there is an option to give a free answer in an “other(s)” category, but typically this type of survey aims at gathering structured and quantitative data. Surveys with multiple-choice answers, rating scales and/or other definite answer fields will provide you with categorical answers that can easily be turned into a visualization or summary for a quick overview of predefined interest areas, e.g. whether the respondents are satisfied or not with the specific service or product you are offering.
Surveys with leading questions, like surveys with definite answers, aim at triggering responses from the respondent about specific interest areas, e.g. whether they are satisfied with the service you provide or what improvements they would suggest for your app. Leading questions are biased towards a specific answer, but contrary to the survey with definite answers, the respondents have space to provide feedback with their own words.
A leading question such as “How can we improve our app?” will lead the respondents to give their opinion about the app and trigger feedback that concerns the weaknesses of the app. Another example of a leading question such as “What do you like about your job?” will trigger the respondents to list all the positive aspects – but not the negative ones – about their jobs. It is important to note that surveys with leading questions will only gather feedback about the interest areas selected in the survey’s design. The data gathered through surveys with leading questions and open answer fields will provide more insights into the respondents’ opinions about the interest areas that are important to the author of the survey, but not what might be important from the respondents’ point of view.
Feedback gathered through surveys with leading questions will be less structured than feedback from surveys with definite answers. However, the answers obtained will typically contain a vocabulary relating to the leading question. Analyzing the answers will provide insights into the most important topics the respondents are talking about in the context of the leading question. This can be analyzed manually by reading the answers and collecting, counting up, and analyzing the topics, or automatically with something called topic modeling. Topic modeling is easily done in Explorer by exploring and modeling all answers to a leading question in topic terms and terms related to them.
Surveys with open-ended questions, like surveys with leading questions, aim at collecting open feedback from the respondents. Contrary to leading questions however, open-ended questions are not biased. The difference between a leading question and an open-ended question is shown below.
Leading question: “What are three things you like about your job?”
Open-ended question: “How do you feel at your workplace?”
The leading question is biased towards a topic and sentiment (“three things you like”), whereas the open-ended question is general and leaves it open to the respondents to give feedback about the positive and/or negative aspects of their workplace. This might seem like a minor difference, however, the data gathered with either of these types of questions differs significantly. While an answer such as “free coffee, office chair, salary” is an acceptable answer to the leading question, this answer is rather odd to the open-ended question. The sentiment is defined in the leading question with the word “like”, so that it is clear that “free coffee, office chair, salary” relate to the positive aspects of the job. In the open-ended question however, respondents will be pushed to answer in full sentences and express their opinions with sentiments. This difference is important if you are planning to do a sentiment analysis of your survey feedback. Sentiment analysis of unstructured data can be extremely insightful; however, it only works if the data contains sentiment.
Surveys with open-ended questions allow respondents to give feedback about the matters that are important to them, which might not have been considered or addressed previously in other sources of feedback. In this regard, open-ended questions are unique in that they can reveal the unknown unknowns, i.e. the strengths and weaknesses you were not aware that you didn’t know about.
Contrary to surveys with definite answers and leading questions, the data gathered through surveys with open-ended questions is unstructured and would generally be described as “messy”. To uncover the unknown unknowns, the method of analysis needs to summarize the data in an orderly manner without missing the important insights. This is the strength of the Explorer. By modeling and exploring the unstructured data, topics and sentiments related to them allow the unknown unknowns to be revealed.
Takeaways
All three types of surveys discussed above have their advantages. The challenge is to create surveys with the questions that will provide the feedback that you are looking for as well as to apply the analysis method that will provide the most insightful results. Below are the most important takeaways:
- If you want to have a quick overview of the respondents’ opinions on selected topics a survey with definite answers is the way to obtain the information. However, you will not receive any or little information about the reason for the respondents’ opinions.
- If your aim is to gain insight into what to improve in a certain area, you might want to choose a survey with leading questions. With questions leading towards topics, you will gather information about the survey takers’ opinions of the leading topic. Their answers will provide you with keywords that can be categorized in action points by means of topic modeling. However, their answers will not be fit for a sentiment analysis. Further, you will not know about what is important to the survey takers, what we call the unknown unknowns.
If you are interested in open feedback from survey takers, a survey with open-ended questions will provide unbiased reviews. By means of exploring, topic modeling, and sentiment analysis of these reviews, the issues that are important to the survey takers about a certain service, product or other interest area as well as potential unknown unknowns will be revealed.
