Transcript Document
Evaluation – the why’s, the
what’s and the how’s
2014
Dr Basma Ellahi (PI)
Cat Crum (PhD Student)
What is evaluation?
• Evaluation is a process that critically examines a
program. It involves collecting and analysing
information about a program’s activities,
characteristics, and outcomes. Its purpose is to
make judgments about a program, to improve its
effectiveness, and/or to inform programming
decisions (Patton, 1987).
Why evaluate?
• Answer the objectives of the
scheme/project
• Demonstrate change due to
the scheme/project
• Determine effectiveness i.e.
causality or association
• Inform improvements
‘dull but important
When compared with the creative and exciting
process of conceiving and initiating a project,
evaluation can often be forgotten and be
perceived as dull! However, some form of
evaluation or formative feedback is the only
thing that will show the effectiveness of the
project
• Evaluations fall into one of two broad categories:
formative and summative.
• Formative evaluations are conducted during
program development and implementation and
are useful if you want direction on how to best
achieve your goals or improve your program.
• Summative evaluations should be completed
once your programs are well established and will
tell you to what extent the program is achieving
its goals.
Type of Evaluation
Purpose
Formative
1. Needs Assessment
Determines who needs the program, how great the need is, and what
can be done to best meet the need. A needs assessment can help
determine what audiences are not currently served by programs and
provide insight into what characteristics new programs should have
to meet these audiences’ needs.
2. Process or Implementation Evaluation
Examines the process of implementing the program and determines
whether the program is operating as planned. Can be done
continuously or as a one-time assessment. Results are used to
improve the program. A process evaluation of a program may focus
on the number and type of participants reached and/or determining
how satisfied these individuals are with the program.
Summative
1. Outcome Evaluation
Investigates to what extent the program is achieving its outcomes.
These outcomes are the short-term and medium-term changes in
program participants that result directly from the program. For
example, outcome evaluations may examine improvements in
participants’ knowledge, skills, attitudes, intentions, or behaviours.
2. Impact Evaluation
Determines any broader, longer-term changes that have occurred as
a result of the program. These impacts are the net effects, typically
on the entire school, community, organisation, society, or
environment.
Adapted from Norland (2004), Pancer and Westhues (1989) and Rossi et al. (2004).
Aims of evaluation
• How has the pilot been received by the different
participants in each of the health communities/
• What have been the successes and the issues –
lessons learned?
• How successful has the pilot been in moving the
health communities towards working on the
basis of the 5 key principles?
• How successful has the pilot been in helping the
health communities meet their objectives?
Evaluation Objectives of the Malnutrition
Prevention Pilot Programme
• Confirming
outcomes and
impact indicators
• Identify unintended
consequences of
implementation
• Inputs and activities
• Generate evidence
base
• Appraising progress
• Identify limiters
• Use self generated
data
Methodology
Impact Evaluation Framework
Theory of change
Logic Model
Linkages between inputs, activities, outputs
and outcomes
Evaluation Plan
• Why?
• How?
• What?
• When?
• Who?
• Where?
Complex Evaluation
Range of activities
- One-off information
and/or taster session
- Promotional stand
- Promotional Campaigns
- Health days
- Demonstrations
- Cooks and eat
- Community Enterprise
- Training courses
• Combination of
evaluation methods
• Qualitative
• Quantitative
The tools of the evaluator
• Quantitative
• Monitoring Information
• Questionnaire/survey
• Experimental evaluation – RCT, case control, cohort etc
• Qualitative
•
•
•
•
•
Observation
Interviews
Focus Groups
Case study
Documentation
Methods
• Semi-structured interviews
(telephone)
• Action Learning sessions
• Audit - baselines and
MUST
• Sampling strategy –
purposive sampling
• Pluralistic model
• Ethics
• Frame of reference
What does good look like?
• Good evaluation is tailored to your
program and builds on existing evaluation
knowledge and resources.
• Good evaluation is inclusive
• Good evaluation is honest.
• Good evaluation is replicable and its
methods are as rigorous as circumstances
allow.
Common dilemmas
• Intellectual property rights / Data
protection and data sharing
• Be conscious of multiple roles
• Follow informed-consent rules
• Respecting confidentiality and
privacy
• Ethics
• Complexity of data collection sites
How do I make evaluation an integral part of my
program?
• Making evaluation an integral part of your
program means evaluation is a part of
everything you do. You design your
program with evaluation in mind, collect
data on an on-going basis, and use these
data to continuously improve your
program.
To build and support an evaluation
system:
Couple evaluation with strategic planning.
Revisit and update your evaluation plan
and logic model to make sure you are on
track.
Build an evaluation culture
What are the benefits?
better understand your target audiences' needs and how
to meet these needs
design objectives that are more achievable and
measurable
monitor progress toward objectives more effectively and
efficiently
learn more from evaluation
increase your program's productivity and effectiveness
10 reasons to evaluate your
project
So you know whether it’s working
So you can be adaptable
To know how things are working
So you’re aware of unintended
outcomes
5. To be able to better communicate the
value of your work
1.
2.
3.
4.
10 reasons to evaluate your
project cont.
6. To focus your work
7. To help look after the people you work
with
8. Build organisational resilience
9. Know why things are working
10. Life is complicated
Make evaluation part of
your program; don’t tack
it on at the end!
Resources
• Magenta Handbook https://www.gov.uk/government/uploads/syste
m/uploads/attachment_data/file/220542/mage
nta_book_combined.pdf
• The Magenta Book: guidance notes for
policy evaluation and analysis http://www.civilservice.gov.uk/wpcontent/uploads/2011/09/the_complete_mage
nta_book_2007_edition2.pdf
Contact details
Centre for Ageing Studies
Faculty of Health and Social Care
University of Chester
Riverside Campus
Chester
Dr Basma Ellahi – Reader in Food and Nutrition
Email: [email protected]
Professor Paul Kingston – Director of The Centre for Ageing Studies, Professor of Mental Health
and Ageing.
Email: [email protected]
Cat Crum (PhD Student – sponsored by Age UK South Staffordshire)
Email: [email protected]