Most surveyors are constantly looking for ways to reduce survey bias that naturally sneaks into their questionnaires. One way to help reduce bias is to randomize survey question answer choices.
Failure to randomize answer choices can result in order bias because there is a natural tendency to select items at the beginning/ending of a list more frequently than those in the middle. On questions where rating or ranking is involved, item order can also negatively impact these measures.
To prevent order bias from degrading data quality, surveyors can deploy randomization on two levels: category to guarantee that lists are randomized and question to ensure items on a page, such as those in a Likert scale, are randomized. This process works to ensure that a given item is selected or rated properly because of its impact and not because of its placement in a question.
Watch a clip from one of our recent webinars, Crimes in Survey Design, to learn more about this survey design crime.
However, also be mindful as you craft your surveys that not all questions should have a randomized answers:
- Demographic questions such as age ranges, education levels completed, etc. When there's a natural, accepted order for questions, you don't need to worry about randomizing the order of your survey answers.
- Long lists of options. Sometimes it's better to sort long lists in some sort of order, such as alphabetically, to reduce frustration and confusion for the respondent. For example, you wouldn't want to randomize a long list of stores in a mall directory because it makes it difficult to quickly find the stores he or she visits most often.
Want to learn about the other crimes of survey design? Watch the full on-demand webinar here!