Transcript Document

Alcohol Improvement
Programme Evaluation
Michelle Cornes &
Michael Clark
Outline plans for the evaluation of the AIP –
What support will be available to Early
Explain the underlying rationale - What is a
‘programme evaluation’ and what do we want
to achieve?
In terms of the ‘how to’ – introduce an
outcomes based approach.
A Co-research Model
Each site will be responsible for designing their own
evaluation strategy and incorporating this as an
‘integral’ part of their Local Improvement Plan.
An evaluation co-ordinator will offer support and
guidance (site visits, telephone, email)
A National Collaborative will act as a forum for the 20
Early Implementer Sites (‘peer support’ and ‘peer
At the end of the Programme, an overview report will
pull together the learning from all 20 Early
Implementer Sites.
Evaluation Timetable (to be confirmed)
Preliminary work on evaluation strategies to
be completed by March 2009
National collaborative to meet every quarter
Each site to bring to the collaborative a short
progress report of their emerging findings
Final evaluation report to be produced by
each site by month 36
Overview report completed 2 months later
Evaluation Requirement
‘[The AIP] will establish a group of early
implementers who will go further a little bit faster in
defining and implementing solutions for tackling
alcohol-related harm. These PCTs will have priority
access to [resources] and their experiences will
contribute to the bank of good practice which will be
disseminated via [a] learning centre to the rest of the
[Accessed 14.8.08]
Evaluation Focus
High Impact Changes
Based on the best available evidence the
Department of Health has identified key
actions that Primary Care Trusts can take
which will make the highest impact on
reducing alcohol related harm and
“Spreading information about best practice is
one thing; the real challenge is to secure
successful exploitation and adaptation of that
Stephenson (2002)
Key questions for the evaluation:
How effective is your ‘Local
Improvement Plan’ as a strategic
change mechanism in reducing alcohol
related harm/admissions to hospital?
Is it helpful/useful to work together
collaboratively in this way? Has the
support offered to you as part of
the AIP been helpful?
Undertake a series of planned actives
Collect evidence in a systematic way
Use more than one method (quantitative/qualitative)
Include multiple perspectives
Make a judgement about the effectiveness of your
local improvement programme
Make a judgement about the effectiveness of
working together in this way
Being realistic
A DIY Evaluation will inevitably be a smallscale piece of work, employing simple
techniques but this is not to down play its
value… It can provide a wealth of interesting
and useful information, and be a positive
learning experience for all concerned.
Why design programmes on the basis of outcomes?
It is what we do (implicitly) already - focus on needs
To more clearly demonstrate impact
To more clearly show added value
An improvement on inputs-outputs approaches
Means of clearly agreeing what we are to do
Making clearer what we can control or only influence
A means to more clearly make and show links - e.g. across
programmes, between national & regional levels
It shows what our ‘products’ & ‘services’ are for
It leads to a clearer understanding of a programme logic
Leads to a better position for monitoring/evaluation
What do we mean by outcomes?
the changes, benefits, learning or other
effects that result from the work that we do.
Control /
Immediate Outcomes
Intermediate Outcomes
Ultimate Outcomes
Measuring outcomes
Hard and soft outcomes
Direct indicators vs indirect indicators
Single indicator vs a basket of indicators
Can we challenge ourselves to measure
Outcomes and the programme logic
Ultimate Outcomes
Intermediate Outcomes
Immediate Outcomes
Programme Logic/
Change management
Education theory
Psychological theory
Social Marketing
This is sensible/plausible
Potential Danger
Limiting ourselves to the more easily
controlled outcomes
Do not limit your ambitions, just be honest
about what is in your immediate control, what
you can influence and what you can lay
down some ground work for but which is
ultimately within the control of others.
Examples of outcomes from DRE
Delivering Race Equality in Mental Health
 Immediate - Links to and activity in all relevant
statutory agencies; work with local 3rd sector and
communities;better local information; more
awareness of personalisation and recovery; safer inpatient care;
 Intermediate - improved access to psychological
therapies; stronger community capacity to lead;
more confidence in communities; increased
 Ultimate - equal access to services; equal outcomes;
better recovery and social inclusion.
Examples of outcomes from DRE
Ultimate Outcomes - priorities and rationale
Definition of
on in
1. Numbers
of service
users by
2. DUP
What the indicator
What the expected
direction of travel
would be
Data source(s) and
levels of coverage
1. We would expect
rates of using
services to reflect
the ethnicity of local
populations and
their incidence of
2. We would expect
DUP to be no longer
for ethnic minority
As services provide
equal access we
would expect this to
influence the
proportions of BME
users to fully reflect
their psychosis
incidence. Overall an
upward trend in the
proportion of BME
users is likely.
1. LDPR (Q4 2008-9)
2. Possibly datasets
from services and
related research (LEO,
Worcester, EDEN,
MiData, FERN)