Surveys, interviews, and focus groups Lorrie Faith Cranor September 2011 CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/
Download
Report
Transcript Surveys, interviews, and focus groups Lorrie Faith Cranor September 2011 CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/
Surveys, interviews, and focus
groups
Lorrie Faith Cranor
September 2011
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
1
Surveys, interviews, and focus groups
Surveys
– Ask people set list of questions (possibly with conditional
questions or branching) with multiple choice or free response
answers
– Can be conducted in person, by postal mail, by phone, online
Interviews
– Ask people open-ended questions, with follow-up questions
depending on their responses
– Usually conducted in person or by telephone
Focus groups
– Group interview, usually with 5-10 participants
– Usually conducted in person
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
2
What method should I use?
Survey
–
–
–
–
Allows you to collect data from a lot of people relatively quickly
Easy to collect data in standardized format, ready for analysis
May be difficult to answer “why” questions
Useful for validating hypotheses from smaller studies with a larger population
Interview
– Allows you to probe mental models, what people think and why
– Semi-structured interviews allow you to explore issues as they are raised
– Allows you to clarify if people don’t understand a question
Focus group
–
–
–
–
–
Similar to interviews, but more efficient as you can interview 5-10 people at once
Less detail from any interviewee than you would get in an individual interview
Not great for testing usability because participants probably won’t all use the software
Sometimes an opinionated individual can dominate a focus group
Hard to publish paper based only on one or two focus groups
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
3
SURVEYS
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
4
Designing good survey questions
Word questions clearly, without jargon or
undefined abbreviations
Avoid leading questions, ambiguous terms, or
emotionally-loaded terms
Design questions to evoke truthful responses
– Non-threatening, don’t bias participants to provide
what they think you want, protect confidentiality
Probe one dimension at a time
Design questions such that respondents are likely
to provide a range of answers
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
5
Multiple-choice answers
Make sure answer choices are clear, mutually
exclusive, cover entire space of possible answers
Make sure answer choices are at appropriate
level of granularity
Where appropriate, allow respondents to indicate
they don’t know, don’t have an opinion, or the
question is not applicable to them
Use consistent rating scales throughout your
survey
Be aware that the rating scale can skew
responses (people like to think they are normal)
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
6
More survey design tips
Cluster similar questions together
Do not ask respondents to perform cognitively
difficult tasks (unless you are testing their
ability to perform these tasks) such as ranking
more than 5 items
Use a clear and attractive layout
Pilot, pilot, pilot!
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
7
INTERVIEWS
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
8
Developing an interview script
Keep your questions fairly open-ended
– You can follow-up with specific probes (“What files do you
have on your computer that you consider valuable?”
Follow up with “Do you have valuable photos? Videos?
School work? Letters?”)
Start with general questions and get more specific so
you get their unbiased impressions before you direct
their thinking to particular details
A semi-structured approach allows you to adjust as
needed
Write-out all your questions and follow-up probes
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
9
Role play and hypothetical scenarios
Appropriate for some interview studies
Give participant a role to play or put them in a
hypothetical scenario
– Imagine you just saw this message on your
computer screen….
– Imagine your friend called you and told you he
saw this message and asked you what to do….
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
10
Preparing for an interview
The day before the interview:
– Print out:
• Protocol – including detailed interview script
• Official consent form
• Payment sheet
– Prepare:
• Compensation payment
• Audio/video recording (devices, batteries, extension cords,
etc.)
• Additional material
– Send an email to the interviewee to remind him/her
of the date/time/place
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
11
During the interview
Explain the purpose of the study (unless you need
to hide that)
Ask to read and sign the consent form
If recording/video taping, turn it on!
Perform the study!
Debrief the person (if applicable)
Ask to sign the signing sheet
Give the payment
Thank the person!
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
12
Coding interview data
Transcribe interview data (yourself or with hired
transcription service)
– Depending on purpose of study, transcribing only selected
quotes may be sufficient
Iteratively review transcripts and create code for concepts
mentioned by participants
As new codes are added, check to see whether those
concepts were mentioned in previously analyzed
transcripts
Keep track of how many participants mentioned each
concept to find concepts that resonate with a lot of
participants
Group similar codes together into categories
Note interesting quotes
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
13
Analyzing interview data
Interview can be formally analyzed through a
coding process
Qualitative approach
– Use concepts and categories to develop theory
(Grounded theory approach, does not start with
hypotheses)
Quantitative approach
– With large number of interviewees (~30), and
questions that ask participants to provide numerical
ratings, quantitative analysis may be feasible
– Important to validate for coder reliability
Can be used to develop mental models
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
14
FOCUS GROUPS
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
15
Planning a focus group
Develop very detailed script to guide you
Pre-screen participants to get a good mix of people
who meet your criteria
Setup audio and video recorders, but don’t make
people feel under surveillance
Helpful to have at least 2 people, a moderator and a
note taker
Give people name tags with their first name only
Plan to do multiple focus groups to mitigate effects of
dominant participant steering conversation
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
16
Conducting a focus group
Make the session fun, informal, relaxed feel
– Provide drinks and snacks
Promote a free flowing conversation that engages
all participants
– Ask open ended questions
– Show people multiple things and ask them to
compare
– Give demos or show videos to start-off discussion
– Give people handouts and ask them to circle things
they like/don’t like, or jot down first impressions
before discussing with the group
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
17
Using videos in focus groups
Videos can provide concrete scenarios for
people to discuss
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
18
Cylab Usable Privacy and Security
Laboratory
http://cups.cs.cmu.edu/
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
19