Transcript Chapter X: Put chapter title here

DECIDE:
An evaluation framework
DECIDE:
a framework to guide evaluation
 Determine the goals.
 Explore the questions.
 Choose the evaluation approach and
methods.
 Identify the practical issues.
 Decide how to deal with the ethical
issues.
 Evaluate, analyze, interpret and
present the data.
Determine the goals
What are the high-level goals of the
evaluation?
 Who wants it and why?
 The goals influence the approach used for the
study.
 Some examples of goals:

 Identify the best metaphor on which to base the design.
 Check to ensure that the final interface is consistent.
 Investigate how technology affects working practices.
 Improve the usability of an existing product .
Explore the questions
All evaluations need goals & questions to
guide them.
 E.g., the goal of finding out why customers
prefer to purchase paper airline tickets
rather than e-tickets can be broken down
into sub-questions:

◦ What are customers’ attitudes to these new tickets?
◦ Are they concerned about security?
◦ Is the interface for obtaining them different, poor, new?

What questions are you asking about the
design of your project?
Choose the evaluation approach &
methods

The evaluation approach shapes the
methods used, and in turn, how data is
collected, analyzed and presented.

E.g. field studies typically:
◦ Involve observation and interviews.
◦ Do not involve controlled tests in a
laboratory.
◦ Produce qualitative data.
Triangulation
Using different methods (or “angles”) to
answer the same or similar research
questions, to verify or refine results
 E.g.

◦ Questionnaire
◦ Interview
◦ Experimental observation
Identify practical issues
For example, how to:
Select users
Stay on budget
Stay on schedule
Find evaluators
Select equipment
Decide about ethical issues

Develop an informed consent form

Participants have a right to:
- Know the goals of the study;
- Know what will happen to the findings;
- Privacy of personal information;
- Leave when they wish;
- Be treated politely.
Evaluate, interpret
& present data
The approach and methods shape the
evaluation, interpretation, and presentation
of the data.
 The following need to be considered:

- Reliability: can the study be replicated?
- Validity: is it measuring what you expected?
- Biases: how is the process reflecting biases?
- Scope: can the findings be generalized?
- Ecological validity: is the environment
influencing the findings? - i.e. Hawthorn effect.
Key points
 Issues to consider before conducting an evaluation
study:
 the goals of the study
 the approaches and methods to use
 practical issues
 ethical issues
 how the data will be collected, analyzed and
presented.
 The DECIDE framework provides a simple checklist
for planning an evaluation study.
http://usableweb.com
This site was created by Keith Instone. It
provides many links to sites that deal
with different aspects of web usability.
There are links to guidelines, examples
of poor design and other information
that will help you to select which
methods to use in your own
evaluations.
http://usableweb.com/topics/000862-0-0.html
provides an alternative way of classifying
evaluation methods.
http://jthom.best.vwh.net/usability/
The Usability Methods Toolbox site,
developed by James Hom,
 last updated in 1998 but it contains many
useful links to sites that discuss usability
evaluation methods
 offers an alternative framework for their
classification.

http://www.otal.umd.edu/hci-rm/index.html
This site offers a classification and
descriptions of some key techniques.
 Graduate Students at University of
Maryland developed this site.
 There is also a section that discusses the
role of theory in HCI design and
evaluation.

http://universalusability.org/
This site groups information about different
methods under three headings:
 testing
 inspection
 inquiry
The information that it provides is very mixed in
quality so be particularly discerning.
Much of the info comes from a limited range of
sources.
Good parts include a link to Gary Perlman's site
on questionnaires and good descriptions of
inspection techniques.