Transcript The role of CAFAS in measuring organizational and system
Performance measurement: Finding our way from outputs to outcomes
Finding our way from outputs to outcomes
How do we know we are having an impact on children, youth and their families? How can we come together to ensure that the services we offer are appropriate and effective?
Panel members
Dr. Melanie Barwick: The role of CAFAS in measuring organizational and system outcomes for children and youth’s mental health Roger Rolfe: The CYTS experience with CAFAS Samantha Yamada: Building research and evaluation capacity at PRI
Overview: Performance measurement
WHAT is it?
WHY is it necessary?
WHO benefits?
How is it done?
What is performance measurement?
“The regular collection of information for monitoring how a policy, program or initiative is doing at any point in time. It can be used to report on the level of attainment of planned results and on performance trends over time.” - Treasury Board Secretariat of Canada
Why is it necessary?
“Serves as a descriptive tool” on how a project, policy or program is doing Serves as an early warning if the direction of a program, policy or project is not going as planned
Who is the audience?
End users: clients, families caregivers Service providers, educators, program staff Organization or network Health system Public at large
How is it different from evaluation?
PM: provides regular snapshots of how a program or policy is doing; focuses on what the outcome is Client satisfaction surveys is often a key indicator of PM Evaluation: can provide insight into how and why an outcome is occurring Client satisfaction is part of the process evaluation and can influence outcomes
Performance measurement and evaluation
SITUATION INPUTS
Resources of a program
ACTIVITIES OUTPUTS OUTCOMES
Quantity of work, products or participants Change in target audience
How is it done?
Program logic models Balanced score cards Strategy maps
Balanced scorecard approach on health promotion
Source: ICES, 2004
Improve client focus of addiction services Further develop & increase equitable resources and capacity Ensure evidence informed practices are developed, implemented and maintained across province Increase productive use & appropriate allocation of resources across system Improve access to appropriate addiction treatment Ensure the continuum of interventions includes prevention, health promotion, early intervention, harm reduction and treatment services. Improve healthy behaviours, health promotion and disease prevention Reducing Risk through influencing the broader determinants of health Increase availability and retention of the qualified human resources Increase linkages, transition & integration within addiction services Improve health outcomes at the individual and population level Ensure quality assurance within the addiction system Improve linkages and transitions between addiction, mental health, health, education, social and justice systems Increase sustainability and equity of the addiction & health systems
Addiction System Strategy Map Ontario Federation of Community Mental Health & Addictions Programs
Child welfare performance measurement
Source: OACAS QA Framework, 2004
Types of performance measures
Outcome measures Intermediate outcome measures Process measures Output measures Input measures
Strategic priorities by the Select Committee
Challenges and Opportunities
Information management capacity within agencies and in the government Selecting measures • Valid and reliable • Relevant, feasible, sensitive to changes • Developmentally and culturally appropriate Collaboration, buy-in and cultures that foster learning
Questions?
Dr. Evangeline Danseco Head of Evaluation and Research 613.737.7600 Ext. 3319 [email protected]
The role of CAFAS in measuring organizational and system outcomes for children and youth mental health
Melanie Barwick, PhD, C.Psych.
Associate Scientist, Hospital for Sick Children Lead Implementer, CAFAS in Ontario (c) Barwick
1
Overview
• Evidence Base for Outcome Measurement 2 • CAFAS Measure 3 • CAFAS Implementation 4 • Ontario Outcomes (c) Barwick
Evidence Base for Outcome Measurement
1 When clinicians are given feedback about how clients were responding to treatment (as expected, normally functioning, failure to respond), they have an opportunity to improve outcomes and reduce deterioration in the patient.
Lambert, Whipple, Smart, et al., (2001) found that they could identify potential treatment failure based on initial level of disturbance and early negative response to treatment. Providing feedback to therapists enhanced outcomes and reduced deterioration . Those identified as potential treatment failures stayed in therapy longer and had better outcomes when feedback was provided to their clinician .
Patient outcomes can be improved if therapists are alerted to treatment response . This is called “outcome management”.
(c) Barwick
Utility of Outcome Measurement
Using outcomes to MODIFY treatment if necessary Assessing outcomes DURING treatment to track client progress (c) Barwick Assessing the outcome of treatment or service
Benefits of Outcome Measurement
Promotes comprehensive approach to treatment Provides balanced view of strengths, weakness, goals Compliments symptom information Identifies risk behaviours systematically Provides information that is useful for formulation Assists in the development of a treatment plan Informs treatment direction and discharge planning Provides services providers with a common language (CYMH, child welfare, Juvenile Justice, Education Provides a tool for advocacy and benchmarking (c) Barwick
Elements of a Successful System of Care in CYMH?
• business practices • human resources • EBP • Best practices • practice change • implementation • CAFAS • BCFPI (c) Barwick
2
Overview of the CAFAS Measure
Provincially mandated use of a the Child and Adolescent Functional Assessment Scale (Hodges, 2002) to measure level of functioning outcomes among 6-17 year old children and youth receiving mental health services in Ontario Begun in 2000 with training of over 3000 practitioners over 3 years; now reaching 6,000 practitioners!
120 CYMH organizations selected by MCYS to participate in the initiative CMHCs also participate in use of a systematic intake screening interview called the Brief Child and Family Phone Interview; oversight and training for BCFPI is provided by Children’s Mental Health Ontario (c) Barwick
AGE GUIDELINES
CAFAS - Child and Adolescent Functional Assessment Scale children ages 6-17 PECFAS - Pre-school and Early Childhood Functional Assessment Scale children ages 4-7
CAFAS Subscales
(c) Barwick
Levels of Functional Impairment
Severe (30) Moderate ( 20 ) Mild (10) None (0) (c) Barwick
SCHOOL / WORK SUBSCALE
Severe Impairment
(30) 001 Out of school or job due to behaviour that occurred at school or on job during the rating period (e.g., asked to leave or refuses to attend. 002 Expelled or equivalent from school due to behaviour (e.g., multiple suspensions, removed from community school, placed in an alternative school. 003 Judged to be a threat to others because of aggressive potential. // 011 EXCEPTION
Moderate Impairment (20)
012 Non-compliant behaviour which results in persistent or repeated disruption of group functioning or becomes know to authority figures other than classroom teacher (e.g., principal) because of severity and/or chronicity. 013 Inappropriate behaviour which results in persistant or repeated disruption of group functioning or becomes know to authority figures other than classroom teacher (e.g., principal) because of severity and/or chronicity. // 021 EXCEPTION
Mild Impairment (10)
022 Non-compliant behaviour results in teacher or immediate supervisor bringing attention to problems or structuring youth’s activities so as to avoid predictable difficulties, more than other youth. 023 Inappropriate behaviour results in teacher or immediate supervisor bringing attention to problems or structuring youth’s activities so as to avoid predictable difficulties, more than other youth. // 027 EXCEPTION
Minimal or No Impairment (0)
028 Reasonably comfortable and competent in relevant roles. 029 Minor problems satisfactorily resolved. 030 Functions satisfactorily even with distractions. 031 School grades are average or above. 032 School work is commensurate with ability and youth is mentally retarded. // 039 EXCEPTION 31
CAFAS Caregiver Subscales
(c) Barwick
Caregiver Resources, Material Needs…
Caregiver difficulties in providing for the child’s material needs - housing, food, clothes - and there is a negative impact on level of functioning Child’s needs for food, clothing, housing, medical attention are not being met, causing Insufficient material severe risk needs leads to frequent negative impact on the child An occasional negative impact due to this depravation 33
Caregiver Resources, Social Support
Caregiver difficulties in providing a home setting that is free of known risk factors (abuse, parental alcoholism) or in providing for the child’s emotional & social needs Caregiver is hostile, rejecting, or does not want child to return to the home Family members are insensitive, angry and/or resentful to the youth Family not able to provide warmth, security, & sensitivity 34
Score Interpretation
Total Score of: Corresponds to clients who are: 0-30 40-70 Likely referred to qualified health professional Likely requires outpatient services 80-100 110-130 > 140 Likely requires outpatient care with additional services of a supportive or intensive nature Likely requires intensive, community based services, although some youths may need acute residential services at some point Very intensive services would be required; maybe in residential or inpatient settings at some point 35
Milestones in Implementation and Uptake
1999 Ministry Mandate 2004 First Annual Report 2009 Focus on Wiki Supports for Practitioners 2000 Rater Reliability Begins for 120 Organizations 2007 Advisory Expands 2010 Possible Implementation of CAFAS v6.0
(c) Barwick 2002 Software Training and Train-the Trainer Begins 2008 CoPs Reach ~300 Endless Possibilities…
3
CAFAS Implementation
(c) Barwick
Implementation Supports
(c) Barwick 38
Annual Reports
(c) Barwick
4
Ontario Outcomes
Table 2.1 Analyzable cases in period 2004 - 2008 Annual Report Analyzable cases Ratio (2008: other years) Ratio (current : previous year) 2008 (fiscal year) 2007 2006 2005 2004 26,974 23,566 18,623 9,634 6,042 1 1.1
1.4
2.8
4.5
1.1
1.3
1.9
1.6
(c) Barwick 40
Total cases submitted by export deadline:
N = 52,423
Next: Restrict to the exporting time-frame Exclude
N = 151
Cases outside the range or with erroneous admission dates Find cases outside of the admission date interval required by last export (01/04/2008 - 31/03/2009)
N = 52,423
Next: Restrict to the reporting time-frame Retain Exclude
N = 24,144 N = 516
Exclude Exclude
N = 638
Cases without T1 or a T14 (Exit) CAFAS evaluation:
N = 419
Find cases closed prior to 01/04/2008 or cases with a T1 evaluation after 31/03/2009
N = 28,128
Next: Restrict to cases within the 6-18 yrs old age range Find cases outside the age range
N = 27,612
Next: Restrict to cases without a pre treatment evaluation (T1) Find cases without a T1 Retain Retain Cases without a T1 but with a T14 (Exit) CAFAS evaluation:
N = 219
Retain Total analyzable cases:
N = 26,974
Cases with T14 and at least one other CAFAS evaluation:
N = 49
Cases with just T14 CAFAS evaluation
Number and Regional Distribution of Mandated Agencies Submitting Data
2004 2005 2006 2007 2008 Region Central East Central West Eastern Hamilton Niagara North East Northern South East South West Total 11 14 14 9 6 10 6 17 20 107 10 12 7 11 14 14 11 12 10 11 14 14 10 14 12 15 16 14 13 15 13 15 16 14 9 3 8 6 16 16 87 9 6 10 6 17 20 107 9 3 9 6 16 17 93 9 6 9 6 17 20 16 17 (c) Barwick 98 9 5 9 6 9 6 11 6 8 5 9 6 8 6 10 6 21 19 19 21 18 21 119 106 115 14 16 14 8 5 8 6 18 19 108 4,580 3,235 2,872 1,990 2,013 2,566 2,241 3,747 3,730 26,974
Gender Distribution of Children and Youth Receiving CMH Services and CAFAS Rating
100 80 60 40 20 0 2005 (N=9,634) 2006 (N=18.623) 2007 (N=23,566) 2008 (N=26,974) Boys 54.3
55.0
56.5
57.2
Girls 45.0
44.0
43.3
42.6
Unreported 0.7
1.0
0.2
0.2
(c) Barwick
Age Distribution
15 10 5 0 6yrs 7yrs 8yrs 9yrs 10yrs 11yrs 12ys 13yrs 14yrs 15yrs 16yrs 17yrs 18yrs 2005 (N=9,634) 2006 (N=18.623) 2007 (N=23,566) 2008 (N=26,974) 3.8
4.1
4.5
5 5.5
6.3
6.6
6.7
6.7
7.2
7.2
7.4
8.1
8.2
8.3
8.3
8.2
8.5
8.5
8.1
8.2
8.4
8.8
8.3
8.7
9 9.1
8.9
10.7
10.2
10.3
10.5
12.7
11.9
11.3
10.9
12.7
12 11.6
11.4
8.4
8.3
7.8
8.4
5.2
5 5.2
5.1
0.9
0.9
0.8
0.9
(c) Barwick
Children with Complex Needs
100% 80% 60% 40% 20% 0% No Yes Unkno wn M issing Data Out o f Ho me P lacement 52.8
16 3.4
27.8
(c) Barwick Child Welfare Invo lvement 55.7
6.6
0.7
25
Complex Needs (2)
100% 80% 60% 40% 20% 0% No Yes Unkno wn M issing Data Diso rder Diagno sis (P resence o f Substance A buse) 66.9
1.4
3.6
28.1
Yo uth With Develo pmental Disability 60.4
4.3
7 28.3
Yo uth With Chro nic M ental Illness 63.7
2.5
5 28.9
(c) Barwick
Severity at Entry to Treatment for Ontario and Regions (2005: N=9,065; N= 2006: N=18,255; 2007: N=23,566; 2008: N=26,974)
0-30 Some need for service 40-70 Outpatient needs 80-100 Outpatient plus extra supports CE CW E HN NE N SE SW TO 22.0
20.1
19.6
25.0
19.8
33.2
26.6
31.1
19.8
13.8
22.8
19.7
19.2
22.2
20.1
29.4
25.0
34.1
21.3
19.7
22.9
20.8
20.4
20.9
20.6
29.7
24.2
31.1
20.3
23.5
22.6
21.6
18.7
22.2
21.0
29.3
23.7
33.3
18.3
21.7
33.9
34.6
35.9
32.9
36.0
39.4
31.1
37.7
36.9
36.9
36.7
38.4
37.3
41.4
37.9
39.1
30.5
34.4
31.2
33.9
(c) Barwick 38.2
38.2
38.1
39.9
37.9
41.8
40.5
40.8
34.8
36.5
38.3
40.0
36.8
40.1
39.6
41.4
41.1
37.8
33.8
36.5
19.4
21.6
21.8
18.9
20.2
15.2
18.8
14.3
19.1
21.1
20.3
23.2
21.1
21.1
21.9
17.2
19.2
14.3
21.4
20.3
20.2
22.1
20.4
22.0
21.6
17.9
18.8
14.6
22.4
19.4
20.2
20.8
21.9
21.4
20.8
18.2
19.0
16.5
21.5
19.9
Severity at Entry to Treatment (2)
110-130 Intensive needs 140+ Very intensive supports CE CW E HN NE N SE SW TO 11.6
11.9
12.0
11.5
11.9
6.3
9.3
5.5
14.7
15.7
11.3
12.2
13.0
11.4
12.0
7.9
9.8
7.1
12.7
12.6
11.1
11.7
12.5
10.9
12.6
7.6
9.4
8.2
13.3
11.3
11.3
10.5
13.4
9.6
12.6
7.6
9.5
7.4
15.6
12.1
(c) Barwick 7.4
5.4
6.7
4.3
5.6
2.6
7.1
3.4
11.4
11.6
6.7
6.7
8.4
5.0
5.8
3.4
5.7
2.9
9.6
9.1
6.5
6.6
7.8
5.6
6.0
2.6
5.8
3.4
8.9
8.2
6.6
6.2
8.5
4.6
5.1
3.1
5.6
3.7
9.8
8.6
Severe Impairment on CAFAS Subscales at Entry to Treatment – years 2005 to 2008
30 20 10 0 2005 2006 2007 2008 School 26.7
26.2
26.2
27 Home 23.2
21.5
20.7
21 Community Behavior tow ards Moods/ Emotions 4.8
4.5
4.1
4.2
8.4
8.1
8.1
8.8
10.3
9.8
10.2
10.3
Self-Harm Substance use Thinking 4.3
3.7
3.6
3.2
5.6
4.6
4.3
4.3
0.9
0.9
1.0
1.1
(c) Barwick
Average CAFAS Subscale Score at Entry to Treatment (T1) by Sex (N for Boys varies between 15,371 and 15,394 and N for Girls varies between 11,437 and 11,455 due to missing subscale scores)
30 20 17 11.9
14.8
12.2
10 4.3
2.5
0 Scho o l 14.2
10.5
14.9
13.7
3.1
5.2
2.4
3.4
2.6
2 Ho me Co mmunity B ehavio r to wards o thers M o o ds/ Emo tio ns Self-Harm Substance use Thinking Boys Girls (c) Barwick
Exit from Services
Retain
N = 14,757
Find cases with a T14 evaluation (Exit)
N =11,038
Cases with T14.
Retain for further investigations
N =11,038
Last CAFAS = T14 Total analysable cases:
N = 26,974
Calculate Last CAFAS evaluation Find evaluations subsequent to Entry CAFAS and retain them Exclude
N = 12,217
Cases with just T1
N = 3,719
Cases without T14.
Retain for further investigation Exclude open cases and cases where all evaluations are dated before T1 or after the close date Retain
N = 1,388
Cases with T14 but still open
N = 9,650
Cases closed with T14
N = 188
Last CAFAS = the most recent evaluation dated before or at the closing date
N = 11,226
Total cases with a Last CAFAS (c) Barwick Exclude
N = 3,531
Time between Entry to Treatment and Last CAFAS
100 90 80 70 60 50 40 30 20 10 0 Cases with Last CA FA S (N=11,226) Cases with T14 (N=11,038) Cases witho ut T14 (N=188) 0-1mth 3.4
3.5
1.1
1-3mths 18.1
18.1
16 3-6mths 24.5
24.4
28.7
6-9mths 17.4
17.4
15.4
9-12mths 10.9
10.8
16 12-18 mths 13.1
13.1
9.6
18-24 mths 6.8
6.8
9 24mths-hi 5.9
5.9
4.3
(c) Barwick
Percentage of CAFAS Evaluations
100 80 60 40 20 0 T1-Last T1-T14 T1 100 100 T2 0.7
0.7
“Look at all the missed opportunities to help me have better outcomes ! “
T3 12.1
11.3
T4 11.7
11.3
T5 5 4.8
T6 10.8
10.5
T7 2.5
2.4
T8 2.6
2.5
T9 1.2
1.2
CAFAS T Value T10 2.4
2.3
T11_T1 3 1 1 T14 98.3
100 T15 3.3
3.3
T16 T17_24 0.6
0.6
0.3
0.3
(c) Barwick
Change in Average CAFAS Total Score from Treatment Entry to Last CAFAS (N=9,663)
65.5
60 70 60 50 40 30 20 10 0 40 39.13
30 Entry CAFAS Last CAFAS Mean Median Mode 0 (c) Barwick
Change in Average Score on CAFAS Subscales from Treatment Entry to Last CAFAS (N varies between 11,098 and 11,113 because of missing subscale scores )
20 15 14.35
13.27
13.77
12.05
10 5 0 8.81
School 8.14
7.31
7.71
3.46
2.31
3.66
1.11
3 2.62
2.01
1.18
Home Community Behavior Tow ards Others Moods/ Emotions Entry CAFAS (c) Barwick Last CAFAS Self-Harm Substance Use Thinking
Absolute Change in Level of Functioning (N=6,721 for 2006; N=9,663 for 2007 and N=10,955 for 2008 ) Not Improved Improved: 2006: 26.1 % 2007: 25.6 % 2008: 25.8 %
No Change
2006: 73.9 % 2007: 74.4 % 2008: 74.2%
16 14 12 10 8 6 4 2 0 <= -80 -70 -60 -50 -40 -30 -20 -10 90 0 10 20 30 40 50 60 70 80 90 100 110 120 130 >= 140 2006 0.3 0.3 0.4 0.5 0.5 1.3
2007 0.2 0.2 0.3 0.4 0.7
1 2008 0.2 0.2 0.3 0.6 0.8
1 1.7 2.7 4.3 14.2 13.3 14.2 12 9.9 7.1 4.4 3.3 2.2
1.6 2.7 4.3
14 13.9 14.9 11.9 9.6 7.1 5.2 3.9 2.8
1.6 2.8 4.3 13.8 12.8 14.6 11.7 9.8 7.3 5.6
4 2.7
1.8 1.2 0.5 0.1 0.3 0.7
1.8 1.2 0.9 0.7 0.3 0.6
1.8 1.5 0.9 0.6 0.3 0.6
(c) Barwick
Severity of Child Functioning for Various Jurisdictions
Author / Source Sample Description CAFAS total score Mean (SD) Entry Exit Diff Effect Size N= 10,955 children and youth served in community and hospital based , 2008 65.50
1 =39.511
39.13
2 =38.987
26.37
0.67
mental health organizations.
, 2007 , 2006 , 2005 , 2004 N= 9,462 children and youth served in community and hospital based mental health organizations.
N= 6,721 children and youth served in community and hospital based mental health organizations.
N= 2,164 children and youth served in community and hospital based mental health organizations.
N=964 children and youth served in community and hospital based mental health organizations.
64.78
1 =39.886
64.60
1 =40.73
64.84
1 =40.13
63.85
38.86
2 =38.398
38.63
2 =39.33
39.48
2 =40.53
37.85
25.92
25.98
25.36
26 0.66
0.65
0.63
0.64
N=11,815 youth referred to public mental health in fiscal year 2002. Of Hodges, 2003 these, N=2,501 had an intake and discharge CAFAS.
, , MATCH Hodges, Xue & Wotring 2004 N=678 children served by Georgia’s Multi who have severe emotional disturbances requiring mental health treatment in a N=125.
residential -Agency Team for Children setting, 64% male and 36% female. 54% Caucasian. Results are for those with an intake and discharge rating, N=5, 638 youths with serious emotional disturbance (score above 50) ages 7-17 years served in community mental health service providers in (c) Barwick 80 135 89.35
1 =32.35
56 99 63.14
2 =38.78
Not reported in manual No standard deviations reported, hence, no effect size calculation.
26.21
0.66
0.73
Summary
(c) Barwick
Benefits for the Kids
Practitioner use of the CAFAS provides a common language and common metric for the CMH system in Ontario (system integration) Systematic assessment of functioning in multiple areas of the clients life is imperative for comprehensive assessment and formulation The systematic measurement of a client’s response to treatment over the course of formulation and treatment has been shown to improve outcomes
Patient outcomes can be improved if therapists are alerted to treatment response
(c) Barwick
Benefits for Practitioners
Triaging for level of risk Periodic assessment of treatment response leads to improved outcomes Outcome data for service planning Increased receptivity and awareness of EBPs outcome management; and Capacity building for EBP implementation and evaluation; Common language & metric services; across children’s sector Advancing knowledge about how to roll out EBPs and support practice change (c) Barwick 60
Benefits for the Provider Organization
Knowing who they serve Matching their client populations with Hiring staff that meets client needs appropriate services Becoming learning organizations Improving capacity to implement other changes Demonstrating their impact Building accountability and pride in service delivery (c) Barwick
Benefits for the CYMH system
Determine – for the first time ever – improve if Ontario children as a function of the services they receive System-wide use of CAFAS builds accountability of the services we provide for the quality
Access to services is only meaningful if services are effective
Provides an evidence-base from which to develop system and organization-level service delivery improvements (c) Barwick
Thank you
(c) Barwick
Utilizing CAFAS Exports & Reports at CTYS
The
CAFAS in Ontario
Quarterly Report • • •
An excellent resource for performance measurement At 40+ pages, it’s long and not readily accessible for management & staff consumption Capturing trends over time requires another tool
Task
: Abridge the Quarterly Report • • • •
Pull key variables & performance indicators from the Report each quarter Enable comparison over time Present data in a simple spreadsheet Append charts to aid interpretation
Solution
:
Agency CAFAS Performance since 2006.xls
Features of the Spreadsheet • • • • •
Key variables are the rows, shown on left-hand side Each column represents data from one Quarterly Report Data is copied from the Report to the spreadsheet Trend symbols added as final column Charts appended as separate sheets
Sample Chart
How the spreadsheet is used
• • • • Presented to senior management each quarter Highlights and key trends noted Required actions taken Spreadsheet is accessible enough to be useful as a tool for teaching management & staff how to interpret quantitative performance data
Reprocessing CAFAS Exports for Program level Evaluation • • •
Agency-wide views don’t reveal what’s happening at the program level Program level results are required for effective evaluation and QI Program level results needed to explain agency-wide trends
Solution
: CAFAS Program Results.xls
• • •
Split the CAFAS Export data set into program sub-samples Reprocess each sub-sample to create program-level reports Assemble results in one spreadsheet to allow comparison
CAFAS Program Results 2009Q3 for 36 mo. period ending Sep 30/09
How we do it....
• • • • Program-level analysis run every second quarter (six month intervals) Each spreadsheet (2008Q1, 2008Q3...) is placed in the same Excel workbook Program spreadsheets are created capturing performance history for each program Each program’s spreadsheet is copied into its own workbook and charts are appended there.
Program-level Performance History:
GROUPWORK Results since 2008
Program-level Chart:
GW Male/Female Improvement Scores
Techie stuff....
• • • • • Client ID#2 field in CAFAS is used to enter program codes (CO, GW,....) Re-run the CAFAS Export modifying the standard filter to include Client ID#2 field Use SPSS to split the export sample into program sub samples Run reports in SPSS for each program sub-sample A big thank-you to Cristina Vlad and the staff at
CAFAS in Ontario
for sharing their syntax files and training us on their use!
Value-added: some examples
1. Monitoring staff data input (compliance) 2. Face validity check on risk profile of program 3. Gender differences in improvement scores in two programs (CO, ERSP) > what do these mean?
Questions?
Roger Rolfe, MEd, RMFT Research & QA Dept 416-924-2100 ext/238 [email protected]
Measuring Hope
Building a Culture of Research and Evaluation at Pine River Institute Samantha Yamada, MEd Co-Founder
Pine River Institute
► 13-19 year olds ► Family-based ► Treatment for substance abuse
1
Outdoor Leadership Experience
2
Residential Treatment
3
Transition
4
Aftercare
Objectives
1.
2.
3.
Describe the Process Findings Lessons Learned
Measuring Hope
► The importance of Mission ► Defining Success ► Asking The Right Questions ► Being Curious
Year 1
Goals
Establish research as a priority for Pine River Data collection
Challenges:
Low enrollment Unstable work environment Lack of resources Competing priorities
Activities
Development of data collection tools Sporadic data collection Focus groups post program First outcomes report (internal document shared at staff retreat) Networking with key partners
Key Lessons Feedback to staff
critical for buy-in
Qualitative data
collection is useful early in program development is
Year 2
Goals
Consistent data collection Build resources Research network
Challenges:
Admin changeovers Competing priorities Lack of resources
Activities
Research Coordinator established Two outcome reports Quarterly feedback of results to staff Partnership with York University PhD Class Grant Application and Award
Key Lessons Collaborate
with a university Have a
staff member
responsible for research
Support from leadership
is critical
Year 3
Goals
Improve data collection Knowledge Translation Research network
Challenges:
Competing priorities
Activities Key Lessons
Joined collaborative NATSAP research study Hired Research Assistant Two outcome reports Monthly meetings with staff for professional development Online data collection Expanding collaborations to additional research projects
Partnerships
are KEY
Regular contact
with staff is essential Even imperfect
research can help
with resource gathering and policy impacts
Personal connections
(e.g. phone calls or in-person meetings) are best for data collection
Decreased Problem Substance Use
Substance Use Pre-Post 100% 80% 60% 40% 20% 0% Intake (n=110) 3 months (n=14) 6 months (n=40) 1 year (n = 14) Sober Social/Occasional Use Episodic Slips Consistent Problem Use
Increased Academic Functioning
Academic Functioning Pre-Post Program Pre Post 70 60 50 40 30 20 10 0 Graduated In school, In school, avg or above avg unspecified achiev.
In school, failing Dropped out, truant Suspended or expelled
Improved Quality of Life and Future Orientation Average Scores for Quality of Life 10 9 8 7 6 5 4 3 2 1 0 Admission Post-OLE Transition School Departure How happy are you with life as a whole?
How happy are you with your health?
How happy are you about the things you want to be good at?
How happy are you about what might happen to you later on in your life?
Qualitative Data
► “I can say now that I no longer want to be part of my old lifestyle but in my last treatment I still did. I can see good things about me and I have goals and believe I can have a future.” Student 2007 ► “My wilderness experience was the most incredible six weeks of the past decade.” Student 2008 ► “I have learned much more about the meaning of my family and friends as well [as] more about myself and others around me.” Student 2008 ► “My child arrived home some weeks ago from treatment at the Pine River Institute…The first few days went well and I pinched myself...Two week passed. School was ‘good’…at six weeks we’re in a rhythm. ‘Life is good,’ – my child’s phrase, not mine. I see a future, I believe it may be bright.” PRI parent
Lessons Learned
► ► ► Qualitative data collection is great early on Buy-in from Have leadership regular meetings is critical to share findings with staff ► Partnerships with universities or other agencies are great ways to increase research capacity ► Explicitly allocate resources evaluation toward research and ► Reporting findings aids in obtaining resources for the organization and impacting policy