Transcript Laboratory studies Lorrie Faith Cranor September 2011 CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/

Laboratory studies
Lorrie Faith Cranor
September 2011
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
1
Laboratory study process







Design study
Submit IRB application and wait for approval
Recruit participants
Screen participants
Schedule participants
Prepare for study
Conduct study
– Instructions to participants
 Analyze data
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
2
Design study
 See lecture on designing experiments
 Some additional issues for lab studies
– How to provide realistic setting and tasks in a lab
environment
– Whether or not to think aloud
– How to observe, record
– How much to pay
– How long
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
3
IRB application
 Most usable security lab studies will be
minimal risk and qualify for expedited review
 However, use of deception may require more
review
– Use of deception must be justified by you and
approved by IRB
– Requires debriefing
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
4
Recruit participants
 To recruit students
– Flyers around campus
– Posts to campus email lists and bboards (misc.market)
– CMU CBDR http://www.cbdr.cmu.edu/
 To recruit non-students
– Flyers on bus stops, grocery stores, libraries, any local business that has a
bulletin board that will let you post
– Posts to Pittsburgh Craigslist
– Asking your friends and neighbors
– Distribute handouts or make announcements at local religious organizations,
school parents’ organizations, athletic organizations, etc.
• We’ve recruited families by handing out flyers at Schenley Oval during kids’ soccer games
• We’ve recruited senior citizens by visiting senior citizen lunch at Squirrel Hill JCC
• Many local organizations offer inexpensive ads in their newsletters (for example Colfax
PTO http://colfaxpto.org/newsletters/
• Some organizations may be willing to recruit members to participate as a fund raiser
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
5
Screen participants
 Decide what your criteria are to participate
and develop screening survey to screen out
those who don’t qualify
 You may also want to use screening survey to
help you balance demographics
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
6
Schedule participants
 Contact participants by email or phone to
schedule their visit to the lab
– CBDR includes an online scheduling system
 Give participants clear directions for getting to
the lab
 Give them a way to contact you if they have to
cancel or get lost finding the lab
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
7
Prepare for study




Make sure you have copies of study materials, consent form, payment record, etc.
Make sure all equipment is setup, working, has batteries, etc. Test it!
Develop a checklist of everything that needs to be setup before a participant
comes into the lab so you can make sure you haven’t forgotten anything
Bring sufficient cash (or other payment) with you
– Make sure you have exact change!
– You can get reimbursed for human subjects payments, but if you plan ahead you can get a
cash advance so you don’t have to layout money

Post signs so people can find the lab
– Weekends and evenings you may have to meet people outside locked door
– In some buildings posted signs routinely get taken down, plan to repost daily (or more
frequently!)

If at all possible, have at least 2 people run study together
– Useful to have someone greet participants and bring them to lab while someone else runs
study
– For some studies its useful to have someone ask questions and interact with participant
(facilitator) while someone else takes notes

Practice, practice, practice!
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
8
Instructions to participants
 Describe the purpose of the evaluation
– “I’m testing the product; I’m not testing you”
 Tell them they can quit at any time
 Demonstrate any equipment
 Explain how to think aloud
– tell us what they are trying to do
– tell us questions that arise as they work
– tell us things they read
 Explain that you will not provide help
 Explain the basic concept of the UI, but not too much
– Can be done in writing or via video for consistency
 Describe the task
– give written instructions, one task at a time
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
9
Cylab Usable Privacy and Security
Laboratory
http://cups.cs.cmu.edu/
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
10