Surveys and questionnaires

A survey reaches many people but only skims the surface

Intuition

A survey is a form many people fill in alone. You get breadth, a snapshot, at low cost. You cannot ask the follow-up question, so you do not get depth.

A survey is a set of well-defined, well-written questions an individual is asked to respond to. It is normally self-administered without the researcher present and can reach large populations by paper, email and the web. In the lecture, a survey is the complete methodological approach and a questionnaire is the actual list of questions, though the terms are often used interchangeably.

Surveys suit measuring attitudes, awareness, intent, feedback about user experiences, characteristics of users, and over-time comparisons. They give an overview or snapshot of a user population and, if well designed, statistically accurate estimates. They are less appropriate for precise measurements or for identifying usability problems in an interface. Practical considerations are the target respondents (purpose, recruitment and sampling), the design (question types, structure) and the mode of administration. The sampling choices are in the previous topic.

Open and closed questions give different data

Compare

Open-ended

Respondents answer in their own words. Better understanding of phenomena and richer qualitative data, but more time and effort, and the wording must be careful or answers become useless. Each answer is analysed separately.

Lecture examples: why did you stop using the Banjee software, how did you feel about its usability, did it let you complete the tasks you wanted.

Closed-ended

Fixed choices, analysed quantitatively. Fixed alternatives restrict what can be said in the findings.

Ordered (logical order, single choice): ranking, Likert scale. Unordered (no order, single or multiple choice): preferences, demographic information.

Exam detail

Sample exam question 18 on open-ended survey responses: they allow participants to express thoughts in their own words, provide richer qualitative data and require more time and effort. They are not easier to analyse statistically and do not limit the scope of feedback.

Structure, delivery and existing instruments

Begin with instructions (how to mark boxes, whether all respondents answer all questions, who qualifies). Give each section a heading and group related questions. Show the path through the survey, making contingent questions clear with arrows, boxes and indenting. For electronic surveys, provide help links and contact details. Keep the length reasonable, and preview the layout on different screens or pages. You may need different versions for different populations.

The mode depends on how you contact the population: a physical visit or postal address suggests paper, an email address or forum suggests electronic. Web tools named: Qualtrics, SurveyMonkey, Google Forms, Microsoft Forms. To deploy online: plan the timeline, design offline, program the online template, test the behaviour, test with a group that is not part of the survey to check the questions are clear, then recruit.

Validated surveys exist in the HCI literature: SUS (System Usability Scale), CSUQ (Computer System Usability Questionnaire), QUIS (Questionnaire for User Interaction Satisfaction) and WAMMI (Website Analysis and MeasureMent Inventory).

Compare

Advantages

• Many responses quickly from dispersed users
• No special equipment for paper versions
• Low cost
• Non-intrusive

Disadvantages

• Sampling is problematic if population size is unknown
• Shallow data
• No immediate follow-up
• Biased for usage patterns rather than facts

Pitfall

“Easier to analyse statistically” belongs to closed questions. Open responses give richer data at the cost of analysis effort.

Recall

Why are surveys poor for finding usability problems in an interface?

They give shallow data, cannot follow up immediately, and the lecture lists identifying usability problems and precise measurement as less appropriate uses.