Consent, paperwork, the session and the SUS

Intuition

The paperwork is the part of a test the participant sees first and the part that protects both sides. The session routine then keeps them comfortable while you collect the data you planned to collect.

Planning the session is about equipment, time and paperwork

In a lab, configure it to your needs: computer or device, and where the participant, facilitator and executor sit. Set up hardware such as eye-trackers, timers and counters. Decide the length of a session for one participant, typically 30 to 120 minutes. For long sessions, warn participants in advance, schedule breaks between tasks (exercise, toilet break, refreshments) and prepare food and water to keep them at ease. Prepare the paperwork.

Informed consent has two parts and starts with an ethics application

Informed consent is formal and signed permission given by the participant to use data gathered within stipulated limits. Preparation begins with the institutional review board or Ethics Approval Committee, and the evaluator or project manager prepares the application.

Exam detail

USyd ethics application: Participant Information Statement (PIS); Participant Consent Forms (PCF); advertisements, letters and emails seeking participants; interview or focus group questions or themes; letters of support or permission from organisations assisting; external researcher declarations (for researchers not affiliated with the University).

Consent in practice: two parts, PIS and PCF. The participant (or guardian) reads both before the session, can ask questions, and signs the PCF before the session. Prepare two copies: one for the participant to keep, one for submission.

Other forms: Non-disclosure agreements, if the developer or customer requires them to protect IP, included when the PCF is signed. Questionnaires: pre-questionnaire before the study, post-questionnaire after. Examples: System Usability Scale (SUS), and Usefulness, Satisfaction and Ease of Use (USE).

The big day has a before, a during and an end

Before: invite the participant in, offer refreshments and paperwork (PIS, PCF, questionnaire), explain the study and check for questions, have the forms completed, optionally interview them about questionnaire responses. During: hand out tasks one at a time, encourage think-aloud (describe actions and why), stop the test if the participant is in distress. Whether to help depends on the purpose of the test; guide users if questions are asked. End: post-session probes, debrief and answer remaining questions and say what you will do next, thank the participant and give a token of appreciation, prepare for the next participant.

A SUS score turns ten ratings into a number out of 100

The Week 6 worksheet scores the System Usability Scale, using a 5-point coding of responses: Strongly Agree 5, Agree 4, Neutral 3, Disagree 2, Strongly Disagree 1.

Try it: worked example

Ratings for questions 1 to 10 are 4, 2, 5, 1, 4, 2, 4, 2, 5, 1. Find the user score.

Show answer and steps

Answer85

1. Odd questions (1, 3, 5, 7, 9) use rating minus 1. Ratings 4, 5, 4, 4, 5 give 3, 4, 3, 3, 4.
2. Even questions (2, 4, 6, 8, 10) use 5 minus rating. Ratings 2, 1, 2, 2, 1 give 3, 4, 3, 3, 4.
3. Add all ten values: (3 + 4 + 3 + 3 + 4) + (3 + 4 + 3 + 3 + 4) = 17 + 17 = 34.
4. Multiply by 2.5 to convert to a scale out of 100: 34 x 2.5 = 85.
5. To get the SUS score for the product, repeat for every user and average the user scores. A score of 68 is average; above is above average, below is below average.

Pitfall

Forgetting to flip the even items is the usual slip: they use 5 minus rating, not rating minus 1. And the total of 40 points becomes 100 only because of the 2.5 multiplier.

Recall

What is signed before a session, and how many copies are prepared?

The Participant Consent Form (after reading the PIS). Two copies: one for the participant, one for submission.