Transcript Measuring the Outcomes of Interventions with Young

Measuring the Outcomes of
Interventions with Young Carers
PRTC Conference, Manchester, 11/10/10
Introduction
•
•
•
•
•
•
Defining terms
Why measure outcomes?
Development of outcome measurement tools for young carers
Use of the tools
Learning so far
Planning considerations
Definitions
Aims the areas of change you intend to achieve through your
project and which stem directly from the needs of your clients.
Outcomes the changes and effects that actually happen as a result
of your work, expected or unexpected, welcome or unwelcome.
The outcomes you hope to see are all the specific changes that will
help you to achieve your aims.
Outcome change the difference between outcome levels at
different points in time. This requires the collection of pre
intervention (baseline) and post intervention data.
Outputs what the organisation does; the services it delivers.
Source: Cupitt, S. with Ellis, J. (2004) Your project and its outcomes
Why measure outcomes?
•
•
•
•
•
•
•
To see if the intervention is making a difference
To identify what is working or not working
Value for money
Learning in order to replicate or revise interventions
Planning purposes
Being responsible and reflective practitioners
Holding ourselves to account
• To satisfy commissioners and grant bodies
The need for robust measures in the UK
Growth in awareness and recognition of young carers due to:
•
•
•
The work of third sector organisations in prompting public and
political awareness;
Identification through research;
Legislation, guidance and policy initiatives that recognise the rights and needs of
young carers and their families.
Census 2001 UK: 175,000 young carers were identified.
350 projects/services across the UK supporting approx 30,000 young carers.
“ A detailed evaluation of the different approaches and their impact on young
carers and their outcomes is lacking”
(National Carers Strategy, HM Government 2008)
Comic Relief/PRTC Grants Programme
2008/11
Programme aims:
1.
2.
3.
4.
5.
Identify and reduce the extent of inappropriate or harmful caring
responsibilities taken on by children;
Produce positive outcomes for children;
Produce useful learning for other organisations considering
similar work;
Identification of an additional aim based on local need (optional);
Work should strengthen families while supporting young carers.
Projects required to provide direct support to young carers under 21
where the intervention was targeted at the most vulnerable young
carers or those with the heaviest caring responsibilities or
under-represented communities or hard to reach children.
The development process
Stage 1:
Identification of items for inclusion in the tools
Stage 2 (1st pilot)
Reducing the number of items
Identifying missing items
Feedback from services
Stage 3:
Refinement of items using principal
component analysis
Stage 4 (2nd pilot):
Validation of the instruments
Obtaining preliminary normative data
Feedback from services
Stage 5:
Design and publication of the tools and
guidance notes re administration
MACA-YC18
Multi-dimensional
Assessment of
Caring Activities
Young Carers
18 item Scale
MACA-YC45
Clinical
assessment
tool
Multi-dimensional assessment of
caring activities (MACA-YC18)
18-item self-report measure to provide an overall summary score
(index) of the amount of caring activity.
Young carers indicate if they do a task: never/some of the time/
a lot of the time and are then scored 0/1/2 respectively.
Index of caring = the total score.
0
1-9
10 -13
14 -17
18 plus
No caring activity recorded
Low amount of caring activity
Moderate amount of caring activity
High amount of caring activity
Very high amount of caring activity
Six subscales: personal care, domestic tasks, emotional care,
household management, sibling care; financial/practical management.
PANOC-YC20
Positive and
Negative
Outcomes of
Caring –
Young Carers
20 item Scale
The positive and negative outcomes of
caring (PANOC-YC20)
20-item self-report measure to provide a score (index) of the subjective
cognitive and emotional impacts of caring.
There are two 10-item subscales: positive and negative impacts.
Young carers indicate if they do a task: never/some of the time/a lot of the time
and are then scored 0/1/2 respectively.
Scores on both scales have a range of 0 - 20.
Higher scores indicating greater positive and negative responses, respectively.
To calculate positive response score Sum items: 1, 2, 3, 4, 7, 8, 15,18,19, 20
To calculate negative response score Sum items: 5,6,9,10,11,12,13,14,16,17
Scores of less than 12 on the positive scale and/or greater than 8 on the
negative scale may be indicative of concern and along with other assessment
processes may indicate support is required for the young carer and/or their
family.
Potential uses of the tools
• Assessments of carers
• Review of progress/interventions
• Evaluation: pre intervention and post intervention (immediate)
post-post measuring (sustainability)
• Research
• Results can be collated to generate scores at three levels:
1) individual
2) project/service
3) programme
Example of use of pre-intervention
data for a young carer
Katie, aged 13
MACA
15
domestic activity
household management
financial & practical
personal care
emotional care
sibling care
3
3
0
1
4
2
PANOC +ve
PANOC -ve
20
4
Example of pre-intervention data at a
project level
MACA
PANOC negative
PANOC positive
Project ‘A’
10.31
4.64
14.81
Project ‘B’
14.87
8.69
11.62
Young carers at Project A have a lower MACA score than those at Project B,
indicating that they were involved in lower levels of caring compared to young
carers at Project B.
Young carers at Project B experienced much higher negative outcomes of
caring than young carers at Project A and their positive scores were also
lower, suggesting that they were experiencing less positive benefits of caring
than young carers at Project A. Low positive and high negative scores on the
PANOC indicate that there is a potential for concern regarding the outcomes
of caring for young carers at Project B.
Examples of pre-intervention data
at a programme level (11 projects)
MACA-YC18
PANOC pre intervention results
113 fully completed tools.
118 fully completed tools.
Individual scores for young
carers ranged from 1 - 26 that
indicates a wide variability in
the amount of care being
undertaken.
Individual scores ranged from
0 - 20 on both the positive and
negative scale indicating wide
variability in the cognitive and
emotional consequences of caring.
Mean MACA score:
12.06 ie higher end of the
moderate band of the scale with
girls (12.82) and boys (10.76).
Mean pre-intervention PANOC
score:
13.34 (positive) and 6.21 (negative).
Hence across the 11 projects there
is not a level of significant negative
outcomes.
Source: University of Nottingham (2009)
Interim Evaluation Report, Comic Relief/
PRTC Young Carers Grants
Programme 2008/11
Feedback comments
“As a worker we found the questionnaires very useful. Especially
when a young carer has completed both pre and post
intervention questionnaires we can easily recognise the
difference if the young carer has any changes. It is very easy to
analyse the data and it helps us to save a lot of time when we
come to do evaluations”.
“The questionnaire has been a very valuable addition to the work
that we are doing with young carers. It has enabled us to
measure the interventions that we make in a much more
systematic way including the impact of caring responsibilities on
the young people”.
“When the young people have completed these correctly they did
show clearly the extent of caring duties and where these were
inappropriate. This information was useful in terms of influencing
the decision of the local authority to provide a care package”.
Challenges
•
•
•
•
•
•
•
•
Administration of the tools and the issue of trust
More ‘paperwork’ alongside other organisational requirements
Ensuring sufficient time and support for completion of the tools
Issues of literacy and language
Young carer resistance or suspicion
Opting out - not completing post intervention tools
Incomplete tools
Project anxiety
Planning considerations
•
•
•
•
•
•
•
•
•
Staff briefing and exploration of any anxieties
Guidance notes
Integrate into existing paperwork and practice
Personalise the ‘Me and My Young Carers’ questionnaire to
align with your project aims and objectives
Workload implications for staff to support young carers
completing the tools
Administration to assist those responsible for co-ordinating and
analysing the data
Troubleshooting
Translation of instruments
Using the results
References
Stephen Joseph, Fiona Becker, Saul Becker (2009) Manual for
measures of caring activities and outcomes for children and young people
available: www.carers.org (young carers page on assessments).
Stephen Joseph et al., (2009) Assessment of caring and its effects in young
people: development of the Multidimensional Assessment of Caring Activities
Checklist (MACA-YC18) and the Positive and Negative Outcomes of Caring
Questionnaire (PANOC-YC20 for young carers’. Child: Care, Health and
Development, 35 (4) p 510-520.
Honour Rhodes (2009) Knowing what you do works. Family and Parenting
Institute.
Sally Culpitt with Jean Ellis (2004) Your project and its outcomes, Charities
Evaluation Service.
For a range of reports and resources on young carers go to: www.saul.becker.co.uk
Contact
[email protected]
or
[email protected]