Planning data gathering: five key issues and recording

Five decisions come before any technique is chosen

Intuition

Running a study without deciding what you want to learn, who to ask and how you will record it gives you a pile of data you cannot use. The five key issues are the planning checklist.

Lecture 4 lists five key issues for requirements and data gathering.

Mechanism

1. Setting goals: the purpose of the session influences the choice of technique and how you analyse the data afterwards.
2. Deciding from whom to gather data: how many participants, and whether all stakeholder groups are needed. Two major types of sampling (below).
3. Relationship with participants: clear and professional, with informed consent when appropriate. A participant information sheet (PIS) gives the purpose of the study, the type of data, how data is reported, stored and disposed of, and who has access. A consent form is signed by the participant or legal guardian and gives permission to collect data and conditions for using it.
4. Triangulation: look at data from more than one perspective, with more than one type of data or method, such as quantitative data from experiments and qualitative data from interviews. It supports data validation.
5. Pilot studies: a small trial of the main study to test questions and analysis, adjusting if the information is not what you expect.

Sampling is probabilistic or non-probabilistic

Compare

Probabilistic

Known probability of someone being chosen.

• Census: a response from every member of the population.
• Random: not everyone is chosen, but the likelihood is known.
• Stratified: an appropriate number of responses from each subset.

Non-probabilistic

Unknown likelihood of being chosen. Used when not strictly random, not seeking a population estimate, or the population is not clearly defined.

• Snowball recruitment: respondents recruit others.
• Self-selected surveys: people click a link on a website or social network.
• Volunteer.

Non-probabilistic sampling is accepted in HCI research, though some social sciences do not accept it, so combine it with other methods. The choice depends on the purpose of the research. Do not forget to collect demographic data. Recruitment starts by defining the target population (age, gender, education, computer experience, users of particular sites or systems), how many people it contains, and how you can contact them (a directory, membership list, social networking group).

Pitfall

Self-selected and snowball samples have an unknown selection probability, so they cannot give a population estimate. That is why they are paired with other methods rather than used to generalise.

Notes, audio and video each trade convenience for cost

Notes, audio, video and photographs can be used alone or together. The slide example asks how long it would take to read a 224-page book, read your notes, look at still-shots, listen to the audio or watch the film, to show that the record changes the effort of analysis.

Compare

Notes

Give the analyst a good overview. Less intrusive and more flexible (write, draw, annotate). Help focus on what matters.

Writing and deciphering is difficult and slow. Easy to lose concentration. Biases creep in. Working with a colleague helps.

Audio and video

Audio plus photographs is a useful alternative and adds context. Short video clips capture activity.

Both need special permission and raise privacy issues. Quality must be ensured (sound level, camera angle).

Recall

Name the five key issues and the three probabilistic sampling types.

Setting goals, deciding from whom to gather data, relationship with participants, triangulation, pilot studies. Census, random, stratified.