Add New Page: You are not allowed to add pages Select section/namespace. New page title.
 

Hack Day User Observation Process

  1. Gather volunteers
  2. Create questions
  3. Test questions
  4. Train observers
  5. Conduct tests
  6. Compile data
  7. Meet with observers
  8. Share results
  9. Sample Questions/Tasks

Gather Volunteers

Test subjects

  • New users building a network (Hackaday Marathon)
  • Non-technical users who wish to administer a network that they built (Field Site Administrators, online survey)

Observers

  • Commotion team

Create questions

Preliminary questions Gather user background information; to gauge prior exposure to technical terminology)

Observation tasks Develop reasonable tasks to check understanding of terminology. Create a note-taking document for observers to use with space to record.

Post-test Questions A chance to get feedback from participants after they've gathered a general impression

Test questions

Do a run-through with 2 or 3 participants in order to refine the wording and adjust the time-frame allotted for different tasks.

Train observers

Observers should know how to:

  • Debrief a participant on the purpose of the test and address privacy concerns
  • Understand and use the observation form to record participant behavior
  • Observe unobtrusively, perhaps with the assistance of audio/visual recording devices
  • Administer the post-test survey
  • Thank the participant for their assistance
  • Save the records/results and pass them to Commotion for analysis

Compute data

  • Calculate success rate
  • How many finished the tasks?
  • How long did each task take on average?
  • Rank list of terminology
  • Which terms were misunderstood?
  • Which terms were not understood at all?
  • Which terms were not preferred?
  • Which terms were understood?
  • Rank terminology intervention strategies
  • Which strategies were successful in presenting/explaining terms?
  • Which strategies were not utilized by participants?
  • Which strategies were unsuccessful, perhaps decreasing understanding?
  • Identify Top 5 Problems

Share results

  • How well-defined is our terminology?
  • How well were participants able to proceed through the tasks?
  • How quickly/accurately did participants complete the tasks?
  • Recommendations to proceed

Follow-up at next Hackaday!

Usability testing is an ongoing process. Evaluate the effect of any changes.

Sample Questions/Tasks

Debriefing

“Hello! Thank you for volunteering to participate in our study. We are testing the usability of our software. We would like to know how well our software defines terms and whether our terminology affects the difficulty of certain tasks. Don’t worry; we’re not testing you! Before we start, can I ask you a few questions to understand your background?”

Preliminary Questions

  • Have you ever used Commotion?
  • Have you ever set up or maintained a router?
  • What is your first language?

Observation Tasks

  • Rename a node
  • Send a specific file to a specific node
  • Connect a new client to an existing mesh network

Observer's Notes

  • Number of tasks completed
  • How much time was spent on each task
  • A space to qualitatively record data on specific usability issues: e.g. sticking points, attitudes/responses
  • Tables with specific terminology/strategies being tested and whether they were successful/unsuccessful

Post-test Questions

  • Which task was the most difficult?
  • What was the best (most useful/most helpful) explanation strategy for a term?
  • Do you have any suggestions for improvement?