Transcript GOMS - Georgia Institute of Technology

(1) Action Analysis
(2) Automated Evaluation
CS 160, Spring 2002
James Landay
April 15, 2002
4/15/2002
1
Hall of Fame or Hall of Shame?
 java.sun.com
4/15/2002
2
Hall of Fame
 Good branding
 java logo
 value prop
 Inverse pyramid
writing style
 Fresh content
 changing first
read
 news in sidebar
4/15/2002
3
Hall of Fame or Hall of Shame?
 Bryce 2
 for building
3D models
4/15/2002
4
Hall of Shame!
 Icons all look
similar
 what do they
do????
 How do you exit?
 Note
 nice visuals, but
must be usable
 What if purely for
entertainment?
4/15/2002
5
(1) Action Analysis
(2) Automated Evaluation
CS 160, Spring 2002
James Landay
April 15, 2002
4/15/2002
6
Outline







Review
Action analysis
GOMS? What’s that?
The G, O, M, & S of GOMS
How to do the analysis
Announcements
Automated evaluation tools
4/15/2002
7
Review
 What is a Serif & what is it for?
 the curly feet at the top & bottom of fonts
 give the eye a line to read by: A Serif Font
 Sans Serif means “without Serif”
 What are some “signals” used w/ text?






4/15/2002
type size
type weight
word spacing
line length
leading (line spacing)
type shifts
8
Action Analysis Predicts
Performance
 Cognitive model
 model some aspect of human understanding,
knowledge, intentions, or processing
 two types
 competence
• predict behavior sequences
 performance
• predict performance, but limited to routine behavior
 Action analysis uses performance model to
analyze goals & tasks
 generally done hierarchically (similar to TA)
4/15/2002
9
GOMS – Most Popular Action
Analysis
 Family of UI modeling techniques
 based on Model Human Processor
 GOMS stands for (?)




Goals
Operators
Methods
Selection rules
 Input: detailed description of UI/task(s)
 Output: qualitative & quantitative measures
4/15/2002
10
Quick Example
 Goal (the big picture)
 go from hotel to the airport
 Methods (or subgoals)?
 walk, take bus, take taxi, rent car, take train
 Operators (or specific actions)
 locate bus stop; wait for bus; get on the bus;...
 Selection rules (choosing among methods)?
 Example: Walking is cheaper, but tiring and slow
 Example: Taking a bus is complicated abroad
4/15/2002
11
Goals
 Something the user wants to achieve
 Examples?
 go to airport
 delete File
 create directory
 Hierarchical structure
 may require many subgoals
4/15/2002
12
Methods
 Sequence of steps to accomplish a goal
 goal decomposition
 can include other goals
 Assumes method is learned & routine
 Examples
 drag file to trash
 retrieve from long-term memory command
4/15/2002
13
Operators
 Specific actions (small scale or atomic)
 Lowest level of analysis
 can associate with times
 Examples





Locate icon for item on screen
Move cursor to item
Hold mouse button down
Locate destination icon
User reads the dialog box
4/15/2002
14
Selection Rules
 If > 1 method to accomplish a goal,
Selection rules pick method to use
 Examples
 IF <condition> THEN accomplish <GOAL>
 IF <car has automatic transmission> THEN
<select drive>
 IF <car has manual transmission> THEN
<find car with automatic transmission>
4/15/2002
15
GOMS Output
 Execution time





add up times from operators
assumes experts (mastered the tasks)
error free behavior
very good rank ordering
absolute accuracy ~10-20%
 Procedure learning time (NGOMSL only)
 accurate for relative comparison only
 doesn’t include time for learning domain knowledge
4/15/2002
16
GOMS Output
 Ensure frequent goals achieved quickly
 Making hierarchy is often the value
 functionality coverage & consistency
 does UI contain needed functions?
 consistency: are similar tasks performed similarly?
 operator sequence
 in what order are individual operations done?
4/15/2002
17
How to do GOMS Analysis
 Generate task description
 pick high-level user Goal
 write Method for accomplishing Goal - may invoke
subgoals
 write Methods for subgoals
 this is recursive
 stops when Operators are reached
 Evaluate description of task
 Apply results to UI
 Iterate!
4/15/2002
18
Comparative Example - DOS
 Goal: Delete a File
 Method for accomplishing goal of deleting a file
 retrieve from Long term memory that command verb
is “del”
 think of directory name & file name and make it the
first listed parameter
 accomplish goal of entering & executing command
 return with goal accomplished
4/15/2002
19
Comparative Example - Mac
 Goal: Delete a File
 Method for accomplishing goal of deleting
a file
 find file icon
 accomplish goal of dragging file to trash
 Return with goal accomplished
4/15/2002
20
Comparative Example - DOS
 Goal: Remove a directory
 Method for accomplishing goal of removing a
directory  ?????
4/15/2002
21
Comparative Example - DOS
 Goal: Remove a directory
 Method for accomplishing goal of removing a
directory
 Accomplish goal of making sure directory is empty
 Retrieve from long term memory that command verb
is ‘RMDIR’
 Think of directory name and make it the first listed
parameter
 Accomplish goal of entering and executing a
command
 Return with goal accomplished
4/15/2002
22
Comparative Example - MAC
 Goal: Remove a directory
 Method for accomplishing goal of removing a
directory
 ????
4/15/2002
23
Comparative Example - MAC
 Goal: Remove a directory
 Method for accomplishing goal of removing a
directory
 Find folder
 Accomplish goal of dragging folder to trash
 Return with goal accomplished
4/15/2002
24
Applications of GOMS
 Compare different UI designs
 Profiling (time)
 Building a help system? Why?
 modeling makes user tasks & goals explicit
 can suggest questions users will ask & the
answers
4/15/2002
25
What GOMS can model
 Task must be goal-directed
 some activities are more goal-directed
 creative activities may not be as goal-directed
 Task must be a routine cognitive skill
 as opposed to problem solving
 good for things like machine operators
 Serial & parallel tasks (CPM-GOMS)
4/15/2002
26
Real-world GOMS
Applications
 Keystroke Level Model (KLM)
 Mouse-based text editor
 Mechanical CAD system
 NGOMSL
 TV control system
 Nuclear power plant operator’s associate
 CPM-GOMS
 Telephone operator workstation
4/15/2002
27
Advantages of GOMS





Gives qualitative & quantitative measures
Model explains the results
Less work than user study – no users!
Easy to modify when UI is revised
Research: tools to aid modeling process
since it can still be tedious
4/15/2002
28
Disadvantages of GOMS
Not as easy as HE, guidelines, etc.
Takes lots of time, skill, & effort
Only works for goal-directed tasks
Assumes tasks performed by experts
without error
 Does not address several UI issues,




 readability, memorizability of icons,
commands
4/15/2002
29
Announcements
 Make sure your web sites are up to date
 I scanned last night and saw lots of material missing
 PowerPoint slides, all assignments, mailto link for team!
 We will start grading these soon
 I will schedule team meetings for Fri. & Mon. to demo
your system (turn in write-up on Fri.)
 I will email a midterm survey today – please fill out so we
can adjust
 PocketPC teams come to my office after lecture to pickup
H/W – THANK YOU MICROSOFT!!!!!
 Questions????
4/15/2002
30
Rapid Iterative Design is the Best
Practice for Creating Good UIs
We have seen how computer-based tools can improve
the Design (e.g., Denim) & Prototyping (e.g., VB) phases
Design
Prototyping
Evaluation
4/15/2002
31
Automated GOMS Tools
 Can save, modify and re-use the model
 Automation of goal hierarchy, method,
selection rule creation
4/15/2002
32
QGOMS tool
4/15/2002
33
CRITIQUE
Hudson et al (1999)
1. Prototype system

in this case with the SubArctic toolkit
2. Demonstrate a procedure (task)
 record events
 apply rules
3. Automatically generate KLMs
4. Semi-automatically generate classic
GOMS models
4/15/2002
34
Automated Web Evaluation
 Motivation
 Approaches
 GOMS inspired models
 remote usability testing
4/15/2002
35
Factors Driving Repeat Visits
Should Drive Evaluation
 High quality content
 Ease of use
 Quick to download
75%
66%
58%
(Source: Forrester Research, 1/99)
4/15/2002
36
Max –
WebCriteria’s GOMS Model
 Predicts how long information seeking tasks
would take on a particular web site
 Automated procedure:
seed with start page and goal page
procedure
reads page
model predicts how long to find & click proper link
load time, scan time, and mouse movement time
repeat until find goal page
 Claim time is directly related to usability
4/15/2002
37
Sample of Max’s Reports
4/15/2002
38
Advantages of Max-style Model
 Inexpensive (no users needed)
 Fast (robot runs & then computes model)
 Can run on many sites & compare ->
benchmarks
4/15/2002
39
Disadvantages of Max-style Model
 Focus on time (much of it download time)
 only 3rd in important factors driving repeat visits
 can’t tell you anything about your content
 doesn’t say anything directly about usability problems
 Robots aren’t humans
 doesn’t make mistakes
 remember, GOMS assumes expert behavior!
 doesn’t account for understanding text
 only tries the best path – users will use many
 Major flaw is the lack of real users in the process
4/15/2002
40
Warning
I am a founder of the
following company –
watch for bias!

The Trouble With Current
Site Analysis Tools
Leave
4/15/2002
Unknowns
 Who?
 What?
 Why?
 Did they find it?
 Satisfied?
42
NetRaker Provides User-centric
Remote Evaluation Using Key Metrics
 NetRaker Index
 short pop-up survey shown to 1 in n visitors
 on-going tracking & evaluation data
 Market Research & Usability Templates
 surveys & task testing
 invitation delivered through email, links, or pop-ups
4/15/2002
43
NetRaker Index:
On-going customer intelligence gathering
 Small number of rotated questions increases response rate
4/15/2002
44
NetRaker Index:
On-going customer intelligence gathering
 Small number of rotated questions increases response rate
4/15/2002
45
NetRaker Index:
On-going customer intelligence gathering
 Small number of rotated questions increases response rate
4/15/2002
46
NetRaker Index:
On-going customer intelligence gathering
 Small number of rotated questions increases response rate
4/15/2002
47
NetRaker Index:
On-going customer intelligence gathering
 Increasing these indices (e.g., retention) moderately (5%) leads
to a large increase in revenue growth
4/15/2002
48
NetRaker Usability Research:
See how customers accomplish real tasks on site
4/15/2002
49
NetRaker Usability Research:
See how customers accomplish real tasks on site
4/15/2002
50
NetRaker Usability Research:
See how customers accomplish real tasks on site
4/15/2002
51
NetRaker Usability Research:
See how customers accomplish real tasks on site
4/15/2002
52
WebQuilt: Visual Analysis
 Goals
 link page elements to user actions
 identify behavior/nav. patterns
 highlight potential problems areas
 Solution
 interactive graph based on web content
 nodes represent web pages
 edges represent aggregate traffic between pages
 designers can indicate expected paths
 color code common usability interests
 filtering to show only target particpants
 use zooming for analyzing data at varying granularity
4/15/2002
53
4/15/2002
54
4/15/2002
55
4/15/2002
56
Advantages of NetRaker
 Fast
 can set up research in 3-4 hours
 get results in 36 hours
 More accurate
 can run with large samples (50-200 users -> stat. sig.)
 uses real people (customers) performing tasks
 natural environment (home/work/machine)
 Easy-to-use
 templates make setting up easy
 Can compare with competitors
 indexed to national norms
4/15/2002
57
Disadvantages of NetRaker
 Miss observational feedback
 facial expressions
 verbal feedback (critical incidents)
 Need to involve human participants
 costs some amount of money (typically $20$50/person)
 People often do not like pop-ups
 need to be careful when using them
4/15/2002
58
Summary
 GOMS
 provides info about important UI properties
 doesn’t tell you everything you want to know about a UI
 only gives performance for expert behavior
 hard to create model, but still easier than user testing
 changing later is much less work than initial generation
 Automated usability




faster than traditional techniques
can involve more participants -> convincing data
easier to do comparisons across sites
tradeoff with losing observational data
4/15/2002
59
Next Time
 Advanced User Testing




Appendix A from The Design of Sites
Gomoll paper
Statistica Ch1, and parts of Ch3
Lewis & Rieman Ch. 5
4/15/2002
60