Creating and analyzing open-response or unstructured survey questions is as much an art as a science.
This type of question serves two main purposes. For the survey researcher, it is a gateway to the thoughts of respondents without the influence of researcher-provided options. For the communications and outreach folks, it provides a vehicle for improving engagement with community stakeholders. The dynamics between these two perspectives create a tension that affects both the question development and the response analysis.
The researcher is generally interested in specifics — List three things you like about product X. The open response question specifies exactly what the responses should address ( given that it is open, however, there is no guarantee that respondents will comply). This narrowing of the range of responses makes the task of coding and analyzing the responses comparatively reliable and efficient.
Being more interested in building engagement and trust, the communications and outreach folks tend to prefer more general, less structured questions — Tell us about your most recent experience with X. Since respondents will feel less constrained in what they can address, their range of responses will be broader. Thus, more categories are needed to capture the respondents’ views, making the task of coding and analyzing responses more difficult and less efficient.
When designing a survey, it’s important to consider the purpose of an open-ended question. Is it purely for research interest, building engagement and trust, or both? Will the data be used to augment the responses to other structured survey questions or to find anecdotal stories and evidence? What is the tolerance for reliability and validity of the resulting data? Answers to these questions will help survey designers construct more targeted, valuable open-response questions.