Transcript Chapter X: Put chapter title here

DECIDE:
An evaluation framework
Text p 348 – 365; Text p 626 - 641
The aims are:
• To discuss the conceptual, practical
and ethical issues involved in
evaluation.
• To introduce and explain the
DECIDE framework.
DECIDE: a framework to
guide evaluation
• Determine the goals.
• Explore the questions.
• Choose the evaluation approach and
methods.
• Identify the practical issues.
• Decide how to deal with the ethical
issues.
• Evaluate, analyze, interpret and present
the data.
Determine the goals
• What are the high-level goals of the
evaluation?
• Who wants it and why?
• The goals influence the approach used for
the study.
• Some examples of goals:
 Identify the best metaphor on which to base the
design.
 Check to ensure that the final interface is
consistent.
 Investigate how technology affects working
practices.
 Improve the usability of an existing product .
Explore the questions
• All evaluations need goals & questions to
guide them.
• E.g., the goal of finding out why many
customers prefer to purchase paper airline
tickets rather than e-tickets can be broken
down into sub-questions:
– What are customers’ attitudes to these new tickets?
– Are they concerned about security?
– Is the interface for obtaining them poor?
• What questions might you ask about the
design of a cell phone?
Choose the evaluation approach
& methods
• The evaluation approach influences the
methods used, and in turn, how data is
collected,analyzed and presented.
• E.g. field studies typically:
– Involve observation and interviews.
– Do not involve controlled tests in a
laboratory.
– Produce qualitative data.
Identify practical issues
For example, how to:
•Select users
•Stay on budget
•Stay on schedule
•Find evaluators
•Select equipment
Decide about ethical issues
• Develop an informed consent form
• Participants have a right to:
- Know the goals of the study;
- Know what will happen to the findings;
- Privacy of personal information;
- Leave when they wish;
- Be treated politely.
Evaluate, interpret & present
data
• The approach and methods used influence
how data is evaluated, interpreted and
presented.
• The following need to be considered:
- Reliability: can the study be replicated?
- Validity: is it measuring what you expected?
- Biases: is the process creating biases?
- Scope: can the findings be generalized?
- Ecological validity: is the environment
influencing the findings? - i.e. Hawthorn effect.
Key points
 There are many issues to consider before
conducting an evaluation study.
 These include the goals of the study, the
approaches and methods to use, practical
issues, ethical issues, and how the data will
be collected, analyzed and presented.
 The DECIDE framework provides a useful
checklist for planning an evaluation study.
A project for you …
• Choose an evaluation study from the list of
three URLs in Links.
• Use the DECIDE framework to analyze it.
• Which paradigms are involved?
• Does the study report address each aspect
of DECIDE?
• Is triangulation used? If so, how?
• On a scale of 1-5, where 1 = poor and 5 =
excellent, how would you rate this study?