Transcript Usability Evaluation Methods

Topic 6:
Usability Evaluation of IA
Applications and
Mechanisms
Azene Zenebe, Ph.D.
Bin Mai, Ph.D.
1
Presentation Outline





Introduction
Usability of IA applications and mechanisms Reviewed
Usability Evaluation: What, When and Why
Usability Specification for Evaluation
Usability Evaluation Methods
ˉ Analytical methods
ˉ Empirical methods


Case Study
Summary
2
Learning Objectives and
Outcomes

After completing this module, you should
be able to:
ˉ Describe the factors that affect usability of
security systems
ˉ Describe the importance of evaluation of
usability security systems
ˉ Prepare usability specification for evaluation
3
Learning Objectives and
Outcomes (Continued)
ˉ perform usability evaluation or testing of
a security system using an analytical
method such as expert inspection
ˉ Perform usability evaluation or testing of
a security system using an empirical
method such as a field study or lab
testing
ˉ Report results of usability evaluation as
well as describe how the results can be
used to make improvement
4
Introduction

Usability of IA application and mechanism Reviewed
ˉ Usability refers to the extent to which a product
can be used by specified users to achieve
specified goals with effectiveness, efficiency
and satisfaction in a specified context of user ISO 9241-11
5
Multi-dimensionality of
Usability

Ease of learning

Efficiency of use
Memorability
Effectiveness
Error frequency and severity
Subjective satisfaction




6
“Usable” Security Systems
can easily and quickly learn a security system
that they have never seen to accomplish basic
tasks
can remember enough to use them later
without major cost
are able to effectively perform and successfully
complete security tasks supported by them
cannot make sever and frequent errors
are satisfied with the interface and functions of
the systems

7
Framework for studying
usability of security systems

four principal components in a humanmachine system
ˉ
ˉ
ˉ
ˉ
TOOL
USER
TASK
ENVIRONMENT
8
Four groups of people
involved in Security systems




Definers provide the policies,
guidelines, and standards
Builders are the real techies, who
create and install security solutions
Administrators operate and administer
the security tools
End-users include home users and
employees who are novice to CISS
9
Usability Evaluation: What,
When and Why




Usability evaluation: whether a security
system is usable for the users
Goal of usability evaluation: identify and
correct flaws associated with ease of use
of a security system
Performed during design and testing (or
post-implementation) phases
Evaluation is iterative – an ongoing
process
10
Usability Specification for
Evaluation


Usability specifications are statements of
required usability characteristics that are
precise and testable
Task analysis provides a more precise
specification of what users are expected
to do in order to accomplish a task
successfully
11
A sample usability specification
- authenticity of a website
Subtasks
Usability Outcomes Expected
Displaying the digital certificate of
the website
A user with at least …previous usage
experience should be able to display the
certificate in a 40 seconds or less, with
no errors, and should rate ‘easy of
finding the menu item/icon for
displaying the certificate’ no less than
6 on a 7-point rating scale.
Determining if the website is
authenticate or not
A user with at least …previous usage
experience should be able to read and
comprehend the certificate information
in a 80 seconds or less, with no errors,
and should rate ‘easy of use the menu
item/icon for navigating the certificate’
no less than 6 on a 7-point rating scale.
12
Usability Evaluation
Methods


Analytical Methods - conduct analysis of
a system’s features with the respect to
their impacts for use
Empirical Methods – collect and use
data from a system’s users. It is also
referred as user-based testing
13
Analytical Methods

Expert’s knowledge stated as a heuristic
rules
ˉ Ten Usability Heuristics by Jakob
Nielsen
ˉ Shneiderman’s 8 Golden Rules of
Interface Design
14
Empirical Methods
What usability evaluators want to know
is what happens when users use the
system
Different techniques are

ˉ Field studies
ˉ Usability Testing in a laboratory
ˉ Controlled Experiments
15
Quick Quiz


What are the main advantages and
disadvantages for analytical methods and
empirical methods?
Come up with two sample scenarios in IA
field where you think analytical methods
should be preferred, and two other
scenarios where you think empirical
methods should be preferred
16
Steps for usability testing







Identify and profile the representative users
Select the setting
Decide what tasks users should perform
Decide how and what types of data to collect
Perform necessary activities before test
session
Perform necessary activities during test
session
Perform necessary activities after test session
17
Usability Testing in a
Laboratory

Validity concerns are associated with the
following questions for lab based testing
ˉ Is the prototype system used in the testing
missing any important features
ˉ Are test participants really the kind of users
who will use the system
ˉ Will actual users do tasks like these
participants
ˉ Will actual users be more distracted in their
offices
18
Using the Results of
Usability Testing

Results need to be looked at and
actionable information regarding
usability problems and issues should
be made for design teams

Provide recommendations to address
the identified problems
19
Automated Usability Testing
Tools

A List of 24 Web Site Usability
Testing Tools
ˉ http://www.usefulusability.com/24usability-testing-tools/

UMD list of usability testing tools
ˉ http://otal.umd.edu/guse/testing.html#
sect3a

Jay Forbes’ presentation about
usability testing tools
ˉ http://www.gslis.utexas.edu/~l385t6rb/
auto_tools.pdf
20
Quick Quiz

Suppose you are testing the usability of
an IDS your company decided to
implement.
ˉ What will be the setting of the testing?
ˉ Who will be the representative users?
ˉ What type of data should you collect?
Justify your answers.
21
Quick Quiz


Among IT managers, business
managers, usability specialists, or
general public, who do you think are the
main users for automated usability
testing tools? Why?
What aspects of a usability study do
you believe can never be automated?
Why?
22
Case

Perspectives: Usability Evaluation


Perspectives is a new approach to help clients
securely identify Internet servers in order to avoid
"man-in-the-middle" attacks
 works with Firefox 3 extension
 Demo
Mission of the Perspectives


detect whether a self-signed certificate is valid
detect the fake security certificate attack and will
warn you
23
Usability Evaluation Design

User Population
 Potential Users: Novice, Intermediate and
Expert in Security and IT
 Targeted Users: Subset of the Potential Users

Context of Uses
 Using the Internet
 Home, free WiFi sites, and/or work
 Quite or Not Quite environment

Tasks: Banking, Shopping, etc.
24
Usability Evaluation Design
Perspectives: evaluating the authenticity
of a public key based on accompanying
signatures and making use of a Browser’s
built-in mechanisms for such evaluation
Requirements gathering

ˉDevelop usability specification

Usability Evaluation
ˉ Using Inspection
ˉ Using Empirical
25
Summary
From this module, reader should take
away the following:

ˉ
ˉ
ˉ
ˉ
Usability is a combination of factors
Usability requires that users understands
the organization policy and rules
There exist frameworks that guide the
usability evaluation
For different stakeholders, the goals of
usability differ
26
Summary (continued)
From this module, reader should take
away the following:

ˉUsability specification is required for usability
evaluation
ˉThere are two categories of usability
evaluation methods
ˉThere existing some tools that automate
usability testing
27
Discussion Topics



What are the advantages and
disadvantages of Inspection method?
What are the advantages and
disadvantages of Empirical method?
Compare and contrast the different
methods of data collection. Describe the
advantages and disadvantages of these
methods.
28
Discussion Topics


How useful are these Heuristics for
security systems? Which of the two is
more relevant to security systems? Are
these methods security systems
dependent?
Is there a heuristics for security system
interface design? Is there a
methodology?
29
Discussion Topics


Describe and discuss scenarios where
a system’s usability is important to one
type of users, while not so important to
another type
What are your opinions regarding the
ideas that, as described by Jay Forbe,
“automated usability testing is too good
to be true”?
30
Project Ideas


Suppose your friend Joe opened an Ebay store online to sell his comic book
collections, what data do you collect to
evaluate his website’s usability?
Suppose a university Registrar Office
hires you to evaluate the usability of its
online registration system. What data
would you collect?
31
Project Ideas



Prepare a sample usability specification
built to track usability of a scenario for
setting a firewall in Windows XP.
Develop a usability evaluation desing to
track usability of an IDS (Intrusion
Detection System)
Design a usability evaluation study for
the latest release of PGP.
32
References
1.
2.
3.
4.
5.
6.
Braz, C. and Robert, J.-M. Security and usability: the case of the user authentication
methods. In Proceedings of the 18th International Conferenceof the Association
Francophone d'Interaction Homme-Machine ACM, Montreal, Canada 2006 199-203
Garfinkel, S.L. Design Principles and Patterns for Computer Systems That Are
Simultaneously Secure and Usable Department of Electrical Engineering and
Computer Science, MASSACHUSETTS INSTITUTE OF TECHNOLOGY, Boston,
2005, 470.
Hoonakker, P., Bornoe, N. and Carayon, P., Password Authentication from a Human
Factors Perspective: Results of a Survey among End-Users. In 3rd Annual Meeting
of the Human Factors and Ergonomics Society, (San Antonio, TX, 2009).
Josang, A., Alfayyadh, B., Grandison, T., Alzomai, M. and Mcnamara, J., Security
usability principles for vulnerability analysis and risk assessment. in Twenty-Third
Annual In Computer Security Applications Conference, (Miami Beach, Florida,
2007), 269-278.
Lazar, J. Web Usability: A User-Centered Design Approach. Pearson, Addison
Wesley, Boston, 2006.
Nielsen, J. Usability Engineering. Morgan Kaufmann, San Francisco, 1994.
33
References
7. Rosson, M.B. and Carroll, J.M. Usability Engineering: Scenario-based
development of human-computer interaction. Morgan Kaufmann, San
Francisco, 2002.
8. Shackel, B. Usability - Context, Framework, Definition, Design and
Evaluation. in Richardson, S. ed. Human Factors for Informatics Usability,
Cambridge University Press, Cambridge, 1991.
9. Shneiderman, B. and Plaisant, C. Designing the User Interface. AddisonWesley, Boston, 2005.
10. Weir, C.S., Douglasa, G., Carruthers, M. and Jacka, M. User perceptions of
security, convenience and usability for ebanking authentication tokens.
Computer & Security, 28 (1-2). 47-62.
11. Whitman, M.E. and Mattord, H.J. Management of Information Security.
Course Technology, Thomson Learning, Inc., Canada, 2004.
12. Whitten, A. and Tygar, D., Why Johnny can't encrypt? In USENIX, (1999).
34