Transcript MSSE Course Name - Georgia Institute of Technology

Saturday March 5
Cognitive Walkthroughs Due Today
 User Testing
 Other Evaluation Mechanisms
 Looking Ahead

1
SEng 5115
March 5, 2005
Evaluation with Users
Big investment -- big potential
 Many issues

dealing with human subjects
 which users? which tasks?
 when in the process?
 what to measure?
 how to measure?

2
SEng 5115
March 5, 2005
Dealing with Human Subjects

Your responsibility to protect subjects
distress, embarrassment
 remind them that you are not testing them


Informed, voluntary consent
understand that they can quit at any time
 explain test in lay terms
 if necessary, there is “equipment failure”


Privacy: anonymity, use of image/voice
3
SEng 5115
March 5, 2005
Which Users? Which Tasks

As close to real users as possible


if real users are scarce, try surrogates
Keep close to the real tasks
may need to shorten some for time
reasons
 may need to provide users with
background information

4
SEng 5115
March 5, 2005
When in the Process?

Early is important
low investment
 time to change


Mock-ups and Drawings are OK

issues in how to handle user choice
Partial prototypes when necessary
 Summary: as early as possible

5
SEng 5115
March 5, 2005
What to Measure

Process data
problems, questions, reactions
 what users are thinking


Bottom-line data
mostly later for usability measurement
 not very useful early in design


Asking users questions?

6
problematic -- users will answer
SEng 5115
March 5, 2005
Thinking-Aloud Method

User asked to think-aloud
ask questions (though not answered)
 explain decisions, identify confusion


Tester records session

avoids interfering as much as possible
• only when test would end otherwise

7
explain to subject that you won’t
answer
SEng 5115
March 5, 2005
Eye-Tracking Testing
Technology to support monitoring
where subjects are looking and for
how long
 Challenge: easy to direct results

avoid thinking out loud
 careful presentation of tasks
 careful design to avoid distractions

8
SEng 5115
March 5, 2005
Alternative Methods

Natural testing conditions
gather performance data
 video-prompted review


Two-person tests


Field studies instead of user tests

9
forces “thinking” aloud
consider deployment, logging
SEng 5115
March 5, 2005
Gathering Field Data
How to get it?
Log high-level actions
 Log low-level actions
 Log problems
 Work products

What to get?
Detailed and statistical usage data
 Example cases

10
SEng 5115
March 5, 2005
Examples

Consider a Word Processor

many alternative solutions for
commands
• toolbars, menus, keyboard shortcuts
relative frequencies of commands
 co-occurrence of commands with
undo
 document statistics

11
SEng 5115
March 5, 2005
Examples

Consider a Website

maps of link traversal rates
• traffic maps

hidden co-occurrence
• web usage mining
errors
 apparent rates of “back” from
destinations
 time on page

12
SEng 5115
March 5, 2005
Different Goals,
Different Approaches

Overall “what’s happening”
general data
 possibly lots of data


Test specific questions


13
targeted data
Always consider issues of user
consent
SEng 5115
March 5, 2005
General Guidelines for
User Testing

Plan ahead of time
what data to record
 what instructions to deliver
 what to do if user “falls off prototype”
 when to provide help, and what help


Know your objectives

14
but never lose sight of the user
SEng 5115
March 5, 2005
General Guidelines
Always have a pilot study
 Get professional help for big studies
 In general, it is better if you aren’t
there

too much bias
 subtle clues
 stay behind one-way glass

15
SEng 5115
March 5, 2005
When You Need
Bottom-Line Data

Need specific measurements
median time for task
 comparison of alternatives


Work out the statistics involved


Focus on one type of test at a time

16
statistics cookbooks
can’t time and use think-aloud
SEng 5115
March 5, 2005
Putting it Together

Critique this test plan

background
• outdoor information Kiosk for 2012
Olympics in New York (we can hope!)
• used by English-speaking Americans
• will provide information on events and
schedules, locations, transportation,
results, maps, and other useful
information
17
SEng 5115
March 5, 2005
Kiosk Test Plan


use prototype kiosk
recruit test users: offer $10 for up to an hour
• post at New York employment and social service agencies



users shown kiosk and 20 minute demonstration video
each user has 3 tasks and the system times them on each
task
finally, groups of 5-10 users will be asked
• which tasks they could not accomplish
• what problems they had with the system
18
SEng 5115
March 5, 2005
Usability Goals and
Measures

Concrete, quantitative measures of usability
learning time
 use time for specific tasks and users
 error rates
 measures of user satisfaction


Comparative usability goals

compare with prior versions or competitors
19
SEng 5115
March 5, 2005
Things to Watch

Goals should be realistic


Many goals go beyond the application UI


100% is never realistic
training, manuals
Testing goals should help improve the UI

detail--not just good/bad
20
SEng 5115
March 5, 2005
Exercise: Setting Usability
Goals

As groups, come up with three usability
goals for your project.
try to come up with markedly different goals
to give broader coverage
 discuss the feasibility of testing these goals

• what is needed for the test
• when in the process?
• how much effort, user preparation/training, etc.?
21
SEng 5115
March 5, 2005
Interface Evaluation

Goals of interface evaluation
find problems
 find opportunity for improvement
 determine if interface is “good
enough”

22
SEng 5115
March 5, 2005
With or Without Users

Users are expensive and inconsistent
usability studies require several users
 some users provide great information,
others little


Users are users


23
cannot be simulated perfectly
Best choice--Both
SEng 5115
March 5, 2005
Evaluation Without Users

Quantitative Methods
GOMS/keystroke analysis
 back-of-the-envelope action analysis


Qualitative Methods
expert evaluation
 cognitive walkthrough
 heuristic evaluation

24
SEng 5115
March 5, 2005
GOMS/Keystroke Analysis

Formal action analysis


accurately predict task completion
time for skilled users
Break task into tiny steps
keystroke, mouse movement, refocus
gaze
 retrieve item from long-term memory


25
Look up average step times

tables from large experiments
SEng 5115
March 5, 2005
GOMS/Keystroke Analysis

Primary utility: repetitive tasks
e.g., telephone operators
 benefit: can be very accurate (within
20%)
 may identify bottlenecks


Difficulties
challenging to decompose accurately
 long/laborious process
 not useful with non-experts

26
SEng 5115
March 5, 2005
Back-of-the-Envelope Action
Analysis

Coarse-grain
list basic actions (select menu item)
 each action is at least 2-3 seconds
 what must be learned/remembered?
 what can be done easily?
 documentation/training?


Goal is to find major problems

27
Example: 1950’s 35mm camera
SEng 5115
March 5, 2005
Expert Evaluation

Usability specialists are very valuable

double-specialists are even better
An inexpensive way to get a lot of
feedback
 Be sure the expert is qualified in your
area

28
SEng 5115
March 5, 2005
Looking Ahead

Next week: Usability Laboratory
Location: Walter Library, room B-26
No food in the lab

Focus: user testing examples



29
external “real” client
SEng 5115
March 5, 2005
Looking Ahead

Heuristic Evaluations

individual evaluations first
• each group member should do an individual
evaluation
• you may use either the “old” or “new” heuristics
from the notes
• turn in the individual lists of problems identified

combined results
• as a group, create a merged list of issues
• turn in that list as the group work product

30
Then, revised prototypes (and user testing
plans) for March 26th
SEng 5115
March 5, 2005