Developing Surveys

Asking the Right Questions the Right Way

Developing Surveys - Asking the Right Questions the Right Way

© iStockphoto

Don't rely on guesswork to find out what customers really think. Create a survey!

If you want to know someone's view on a particular topic, you go and ask them, right? Getting their answer directly is much better than second-guessing what their response might be.

The challenge comes when you want to gauge the opinions of several groups of people. For example, you may need to assess the mood of your team or department in response to a new initiative that you've introduced. Or you may want to test out an idea with your customer base before launching a new product or service.

Whether it's corporate culture issues, or customer satisfaction research, the best source of feedback and suggestions is usually the primary stakeholders themselves. We can try to guess what customers want, or what motivates workers... or we can simply ask them directly.

That's where surveys come in. Surveys give you the chance to ask a target group of people a set of questions about a particular subject, project or issue. You can then use their views to tailor your strategy or review your approach.

However, do bear in mind that effective surveys involve much more than writing down questions and asking people to respond.

What, therefore, do you need to know about surveys before you implement one? And how do you maximize the value that you gain from them?

This article outlines the strategies and techniques that will help your survey deliver the desired outcome.

Planning the Survey

If you simply sit down and start writing questions, you may risk missing critical elements of the data that you want to capture. Take time to plan your approach, and decide what you want to achieve.

Consider the following when planning your survey:

  • Determine your survey goals – What do you want to learn from your respondents? The clearer you are about what you need to know, the more focused your questions and answers will be. Do you want satisfaction ratings, or people's opinions? Are you measuring attitudes or predicting future needs?
  • Identify your target population – Who do you want to survey? Respondents could be past customers, potential customers, job applicants, workers in a particular geographic location, and so on. Before you define your target audience, assess how their answers will help you to achieve your objectives. Also, make sure that you have access to your desired respondents. Are you able to contact the right people to ask your questions?

    Be aware of the potential bias associated with your target population. For example, if you survey only former workers, or people who have contacted your customer service department, you'll likely receive a higher proportion of negative answers. This is because former staff may have left the company because they were unhappy, and people tend to contact customer service only when they have a complaint.

  • Decide your sample size – How many responses are enough to give you the information and insight you require? Larger samples usually allow you to reach more reliable conclusions. However, you don't want to survey too many people. Ideally, you need a balance between the number of responses, and your ability to process and analyze them.
  • Choose your survey method – How will you deliver the survey? The time and budget that you've got available will often determine how you carry out the research. Here are some options, and the common advantages and disadvantages of each:
    • Person to person – In one-on-one interviews, it's easy to change or clarify your questions as needed. However, these interviews take a lot of time and resource, so you have to decide if the investment is worth it. Some companies use telephone interviews to reduce costs, but this can also be expensive.
    • Mail – This can be cost-effective and relatively easy to organize. This method also allows respondents to complete the surveys at their own convenience. However, there is a tendency for people to ignore this kind of survey, and there's no opportunity for you to explain your questions. However, there are ways to improve response rates, which we'll discuss below.
    • Internet – Email and online surveys are growing in popularity, partly because they're inexpensive to deliver. Email surveys can be limited in the types of questions you can ask – but you can set up surveys on websites so that they're interactive, and you can even program them to react in real time to the responses given. However, there are disadvantages to internet surveys. Respondents must have access to a computer and the internet. And, you need to make sure that only the target audience has access to the survey, and that respondents answer only once. However, there are various survey tools available – such as SurveyMonkey, PollDaddy and Google Forms – that aim to make the whole process easier.

    Your survey method may create biases. As we said, online surveys limit your respondents to computer users with internet access.

    One-on-one interviews can be affected by the time of day – for example, if you're surveying office staff during work hours, they may be pressed for time. And interviewing people in one location may reach only one socioeconomic group. Consider any potential biases when choosing your method.

  • Determine your survey length – How many questions will you ask? The key here is to be very focused on your goal. If you allow multiple stakeholders to contribute, you'll quickly end up with a list of questions that's far too long. It's important to be ruthless in challenging the value of each question, and to avoid including the vague "nice to knows," or the ones that do not contribute to your goal. Bear in mind that the survey length may influence the quality and quantity of responses that you receive. When a survey is too long, people may get bored – and either fail to complete it, or answer too quickly to save time. If the survey takes more than five minutes to complete, people may not bother doing it at all. If the survey is too short, you may not get enough information to meet your research goals. Also, consider asking the same question in different ways to evaluate the reliability and integrity of the responses. KISS (Keep it Simple, Stupid) is a good guideline when determining your survey length.
  • Choose the types of questions – Will you use true/false, multiple choice, ranking, or open answers? The more customized your responses, the more work it will take to interpret the results. Simple multiple-choice or limited-choice questions can be scanned into a computer, and analyzed quickly. However, you may lose some of the detail by taking this approach, so consider the pros and cons of each option.

It's important to consider these basic planning elements, and be clear about your objectives and basic strategy, before you start writing the survey questions.

Developing the Questions

You can use two basic types of questions in a survey:

  1. Multiple choice – Respondents choose from a series of answers that you provide. This category includes ranking and true/false questions.
  2. Open-ended – Respondents answer with a number, or with a written or verbal response.

Here are some useful guidelines for writing questions:

  • Allow for every alternative – To find out what kind of green vegetables a person regularly eats, it would be difficult to list every possibility. Instead, you could list the more common vegetables, and include a space for "Other." Be careful to allow respondents to include more than one choice. If people eat broccoli, spinach, and green beans, for example, don't force them to choose just one.
  • Allow for Other, None, Don't Know, and Not Applicable responses – Don't force a choice when it may not be relevant. This is also an opportunity to list only the choices most relevant to your objective, and then analyze the remaining "Other" responses separately.
  • Ensure that options are mutually exclusive – If questions force a choice, make sure each option is clear and different. For example, when asking if workers are full time, part-time, or seasonal, remember that seasonal workers can be full time or part-time.
  • Make answer choices specific – You ideally want the most accurate responses possible. For instance, if you ask people if they've ever listened to country and western music, they may answer "yes" – even if they heard only one country song 20 years ago. It's therefore vital to be specific about what you want to know.
  • Ask one question at a time – Avoid questions that deal with more than one issue. For example, consider "Do you think people should be allowed to vote early, and have you ever done so?" If people answer "no," does that mean they disagree with early voting, or that they've never voted early?
  • Limit your choices – Too many options may overwhelm or frustrate respondents, and they may not read the whole list before making a choice.
  • Use an appropriate Likert scale (ordinal scale) – Your response scale should match the way your choices are worded. A Likert scale asks respondents to indicate how much they agree with a statement. Consider, for example, "I often seek other opinions before making a decision." It doesn't make sense to offer the choices Very Often, Fairly Often, Sometimes, Almost Never, or Never – because the statement already includes the word "often." More appropriate choices would be Strongly Agree, Agree, Disagree, Strongly Disagree. An alternative way of doing this is to keep the original responses, but change the statement to "I seek other opinions before making a decision."

    Other response scales you might like to use could include:

    • Excellent, Very Good, Fair, Poor.
    • Completely Satisfied, Very Satisfied, Somewhat Satisfied, Somewhat Dissatisfied, Very Dissatisfied, Completely Dissatisfied.
    • Definitely True, True, Don’t Know, False, Definitely False.
    • None, Very Mild, Mild, Moderate, Severe.
  • Don't "lead" the respondent to answer a certain way – Avoid wording that assumes you'll receive a particular response – and don't make your preferred answer obvious. Consider the question "Isn't it best for animals to live in the wild instead of a zoo?" This will probably lead people to answer "yes." A better question would be "Where is the best place for animals to live?"
  • Be aware of cultural issues – People in some cultures are more likely to be agreeable, or aim to create a positive image through the answers they give. Some cultures teach people to give the answer they think is "correct." Also, remember to take into account the issue of anonymity. You may want to reassure all respondents that their individual answers will be kept completely confidential.
  • Ask clear questions – Above all, questions must be clearly understood. Be aware of these potential problems:
    • Avoid words that may cause certain emotions. For example, words like patriotic, communist, victim, feminist, and insurgent may cause bias in the responses.
    • Avoid business jargon, acronyms and technical terms.
    • Avoid slang and idioms.
    • Avoid double negatives, such as "Should we not appeal the decision not to introduce a dress code?"
    • Avoid ambiguity and vagueness, such as "We should try to limit access to cigarettes most of the time."

Designing the Survey Elements

Formatting and designing the survey will also affect its impact. Consider these tips to maximize the survey's effectiveness:

  • Give your survey a title.
  • Include a short introduction.
  • Include information about you and your objectives, where possible.
  • Guarantee anonymity and confidentiality, if possible. Or describe how results will be shared and ask for agreement.
  • Provide detailed instructions on how to return the survey. Supply pre-addressed, postage-paid envelopes for mailed surveys. Provide contact details where appropriate.
  • Ask for demographic information (age, education level, income level, and so on) at the end of the survey.
  • For written responses, provide enough space to answer open-ended questions.
  • Allow respondents to comment on the survey at the end. Provide space to write responses, or make sure interviewers ask for feedback.
  • Keep lots of white space (areas with no printed text).
  • Ask pleasant and easy-to-answer questions at the beginning. Save the more sensitive or difficult questions for later in the survey.
  • For agree/disagree scales, offer the positive answers first.
  • Mix up the response choices. For example, make sure a"disagree," or other negative response to some questions, actually indicates a positive behavior or perspective. This keeps respondents from getting into the habit of always giving an "agree" answer.

Improving Response Rates

The best-designed survey in the world may not generate the number of responses you need. This is because people who receive surveys often don't care enough to complete them. Some of us receive too many surveys – letters in the mail, phone calls during dinner, and pop-up screens on our computer when we're working. The smart way to carry out a survey involves a variety of methods to persuade people to respond.

Finding This Article Useful?

You can learn another 147 communication skills, like this, by joining the Mind Tools Club.

Join the Mind Tools Club Today!

Here are three ways that can help you improve your return rates:

  1. Design the survey well – Follow the design tips above to improve your response rates dramatically. People are more likely to answer surveys that are well organized and easy to understand.
  2. Keep the survey length reasonable – Don't make your survey too long. Few people want to answer page after page of questions.
  3. Create a perceived value to the respondent – One of the best ways to increase responses is to prove that the information provided will benefit the respondent. For example, if you ask for their input, tell them how their suggestions will be used to impact their work. When you link responses to a positive outcome, you help provide clear motivation for them to respond.

    The other way to add value is to provide a tangible incentive or reward. Even a small token of appreciation may help increase your response rate. And incentives also tend to improve the value of the responses – you typically get fewer "Other" and "Don't Know" answers.

    Finding an incentive that will appeal to your target audience is key to your success. Here are some of the more common ones:

    • Tokens – You might include a small, inexpensive item with the survey, such as a pen or notepad. These are known as pre-survey incentives.
    • Coupons – You could provide coupons with the survey package or mail them out after a completed survey is received. Typically, coupons are for the next purchase from your company. However, you can send coupons for anything you think the respondent might want.
    • Sweepstakes – Enter all completed surveys into a draw for a major prize. You can award trips, free merchandise, or even cash.
    • Other post-survey tangibles – These may include tickets to shows, dinner certificates, spa certificates, and so on.
    • Refunds or rebates – For customer satisfaction surveys, you may offer the chance to win back the purchase price of the product the customer has just bought.

Remember to issue reminders after sending out the initial survey. A postcard that arrives a week after the survey may be enough to motivate someone to complete it. Reminders may not be as effective as rewards, but they can significantly improve response rates.

Key Points

Surveys are a great way to discover what people think. If you need to know something, asking is more efficient than guessing – but you must ask the right questions in the right way. The following four steps can help you design a clear and effective survey:

  1. Plan your survey. Determine your key objectives, target audience, sample size, survey method, survey length, and the types of questions to use.
  2. Develop your questions. Choose between multiple choice or open-ended, but make sure that questions are specific, unbiased and allow for alternative answers.
  3. Design the survey elements. Format and lay out your survey so that it is clear and easy to use.
  4. Improve response rates. Achieve this by using a good design, getting the survey length right, and creating a perceived value to the respondent.