Transcript Slide 1
ARE WE DOING ANY GOOD?
ARE WE DOING ANY GOOD?
Measuring, Monitoring and
Evaluating
Medway and Blyth
Comparing the two….IMD
Benchill
NW
1
Hertsmere
East
8414
Blyth
NE
342-455
Medway
SE
1181-1472
Cowpen
Croft
Kitty Brewster
Twydall
Holcombe
Town
% 11-18
smoking
= 37%
% 11-18
smoking
= 43%
In the past….
Health interventions usually small
scale + local
Evaluation often driven by need to
obtain + retain funding
Few lessons for wider use in policy +
practice
More recently….
Growing interest in community-based
interventions
New opportunities for exploring new
evaluation approaches + outcome
measurement
Culture of accountability
So what is Evaluation?
Evaluation is the systematic collection
of data that will allow a judgement to
be made about the value of a
programme or intervention. The
evidence from an evaluation should
allow practitioners to make suitable
changes to the programme to make it
more effective for participants.
HDA 2001
So what is Evaluation?
Evaluation is the systematic collection of
data that will allow a judgement to be
made about the value of a programme
or intervention. The evidence from an
evaluation should allow practitioners to
make suitable changes to the programme
to make it more effective for participants.
HDA 2001
Assessing whether projects are
effective and acceptable
Nottingham HAZ 2000
Example – the FAB Fruit and
Vital Veg Project
Questionnaire pre and post the intervention
Measured consumption, knowledge and
attitudes to fruit and vegetables
Results:
Average fruit+veg per day
Like veg
Like fruit
Aware of 5-a-day
Pre
Post
1.62
73%
94%
35%
2.30
84%
96%
80%
And also….
Needs supportive teacher
School needs to be ready and committed to
the project
Need a mix of activities including:
Art and drama to help promote image of fruit
and veg
Tasting sessions to increase confidence in trying
fruit + veg
Work with parents on packed lunches
Work with school cooks and caterers
etc etc.
Evaluations give….
Indicators and measures
…of success and change
…of what worked and what didn’t
…and a rich picture of ‘the intervention’
Evaluation should….
Measure effectiveness, efficiency and cost benefits
Measure progress towards aims and objectives –
short and long term
Inform and improve decision making
Give credibility
Inform funding decisions
Give deeper understanding
Develop the skills of those involved
Develop a systematic evidence base
.…be enjoyable and fun!
Where do we start?
Setting the Baseline…
Identify objectives for the
research – our evaluation
questions
Establish
Baseline
Position
Review
Identify indicators and
measures, and how these are
to be collected
Establish current position
(baseline) - using data already
available or data you collect
Continuous review will allow us
to ensure that these measures
continue to be appropriate as
the programme develops.
Identify Appropriate
Measures
and Indicators
Identify Objectives of Research
Components of an Evaluation
BASELINE
Review
PROCESS
EVALUATION
Effectiveness of
planning and delivery,
acceptability,
accessibility etc
IMPACT
EVALUATION
Immediate benefits
OUTCOME
EVALUATION
Longer-term benefits,
sustainability
A Formative Approach
Improvements
Baseline
Learning
Evaluation
The worries….
It’s there
to help us
To make
the project
better
Make
evaluation
the central
element
We all
own the
evaluation
We’re not
being
assessed
Sources of help
Various evaluation
resource packs
Other projects
experience
Health Development
Agency
NICE
http://www.nice.org.uk/
…and many references to other
work
Example – School-based
project
Project – Active programme promoting lifestyle education in
schools (APPLES)
Intervention – project team assisted schools develop action
plans, trained teachers and provided resources
Objective – to implement and evaluate a school based
intervention designed to reduce risk factors for obesity
Design – Randomised controlled trial (RCT)
Sample – 10 primary schools in Leeds – half received the
programme, half were a control and didn’t
Output measures:
Teachers evaluation of training and other input
Success of school action plans
Content of school meals
Children’s knowledge of healthy living and self-reported
behaviour
The Results….
The Results….
And also….
Children who had received
programme showed greater
understanding of health benefits of
diet and physical activity
Children reported changes in
behaviour
Positive changes in playground
activities and tuck shops
Things to remember….
Plan clearly at the start
Evaluation doesn’t need to be difficult or
intimidating, but does need thought and planning.
Build it in as a central item as early as poss!
There is no ‘one best way’ to do an evaluation
Include stakeholders – they will have contradictory
views sometimes – but it’s worth it!
Choose most suitable data collection methods for
answering your evaluation questions
Use Qual and Quant
Document all lessons learnt
Share your findings
Thank you
[email protected]