Transcript Document

READY - A Youth
Development
Outcomes Measure
UNIVERSITY OF
ROCHESTER
MEDICAL CENTER
Project Team

United Way of Greater Rochester
–
–

Rochester-Monroe County Youth Bureau
–

Chris Dandino
Univ. of Rochester Medical Center, Div. of Adolescent Medicine
–
–
–
–
–
–

Kathy Lewis
Elizabeth Ramsay
Jonathan D. Klein, MD, MPH
Melissa Matos Auerbach, MA
Shannon M. Smith
Sheryl Ryan, MD
Cheryl Kodjo, MD, MPH
Premini Sabaratnam, MPH
Representatives from youth serving agencies in Rochester
Participating Agencies included:













Baden Street Settlement
Big Brothers, Big Sisters of Greater Rochester
Boy Scouts of America, Otetiana Council
Boys & Girls Club of Rocehster
Center for Youth Services
Charles Settlement
Community Place of Greater Rochester
Girl Scouts of Genesee Valley
Hochstein Music School
Metro Council for Teen Potential
Southwest Area Neighborhood Association (SWAN)
Urban League of Rochester
YMCA of Greater Rochester
Youth Development




Philosophy or approach - a set of principles
emphasizing active support of the growing
capacity of young people by individuals,
organizations, and institutions
Characterized by a positive, strength building
orientation
Occurs at home, in school, among peer groups,
and in community-based programs
Has gained importance nationally, in states, and
in local communities
Community Outcomes

Local funders are increasingly concerned with
demonstrating effective progress toward
outcomes

Existing measures of youth development are
lengthy and complex
Youth Development Outcomes
Measurement Project
 GOAL:
–
To develop an evaluation tool for YD
programs that met the following
criteria:
 Easy
to Use
 Easy to Administer
 Applicable to a Variety of Youth Development
Programs
 Useful for Assessment of Impact of Program on
Youth Development of Participants
Rochester, New York






City in Monroe County,
Western New York
Population 219,773
52% non-white
37% ages 19 or less
Person under age 18 in 34%
of households
32% of families with children
under 18 below poverty level
Source: US Census, 2000
Youth-Serving Agencies
Serve the Youth of Rochester through:








case management
counseling
homework assistance
sports programs
life skills building
leadership programs
music lessons
provision of safe, open recreational spaces
Three Phase Project

Phase I Instrument development via a
consensus process

Phase II Piloting to test validity and reliability
of instrument

Phase III Field tests and dissemination
Phase I: Instrument Development Dec. 2000 - May 2001

Meetings with representatives from youthserving agencies and funders

Identification of core and optional outcome
measures and questionnaire items

Establishment of face validity of core measures
and measurement strategies
Identification of Core Outcomes

Three meetings resulted in list of 54 indicators and 10
outcomes

peer and adult relationships
constructive use of leisure time
basic social skills
community service
health maintenance
decision making process
responsibility
understanding boundaries/rules
positive identity
independent/daily skills









Narrowing List of Outcomes
Agency representatives were asked:

What impact does your program have?

What would you like to learn to improve the quality of
your program?
Programs Wanted to Know






Effectiveness of staff
Effectiveness of services they provide
Impact on youth and their families
Impact of youth involvement in more than one
program
Youth Development philosophy of staff
Gaps and what programs can do about them
Consensus Process

Programs used nominal iterative process to identify
consensus priority areas for youth that they could
impact

First Round:
18 constructs

Second Round:
7 constructs
–

Top 2 retained
Third Round:
–
Top 2 retained
4 constructs
Outcomes for Operationalization

Basic Social Skills

Caring Adult Relationships

Decision Making Process

Constructive Use of Leisure Time
Candidate Questions
Questions adapted from instruments by






Add Health
Boys and Girls Club of America
Girl Scouts of America
Metro Council for Teen Potential
Worcester Youth Development Initiative
YMCA
Phase II: Piloting the Draft
Instrument - May 2001 - March 2002
Piloting in two phases:
A.
Cognitive interviews to test validity of items
B.
Field test of internal consistency of items and
feasibility
A. Cognitive Interviews







48 urban and suburban adolescents aged 10 to 17
Mean completion time: 11 minutes
70% had no suggestions
67% reported survey was “easy” to complete
81% understood everything in the survey
98% did not mind answering the survey
Items re-worded to increase readability (now at 4th grade
level) and to simplify concepts
B. Field Test




389 urban and suburban adolescents
Ages 10 to 19
Large drop-in programs and smaller, structured
programs
Findings:
–
–
–
–
Feasible for program staff to administer
Large groups required more staff time
Easy for older adolescents
Some issues remain for younger adolescents
Field Test Results




Youth more attached to programs did
better on measures
Four constructs have several good factors
for program use in evaluation
Instrument consists of six factors,
corresponding to three outcomes
Internal reliability scores (as from .5782 to
.8557)
Factor Analysis: Core Outcomes
Self Control
Empathy
Communication
Basic Social Skills
Staff Relationships
Program Effect
Caring Adult Relationships
Decision Making
Decision Making
Constructive Use of Leisure Time
Factor Analysis - all participants
ROTATED COMPONENT MATRIX: ALL CASES (n=389)
OUT C OM E
C ON ST R UC T
IT EM S
Usually feel connected
Listen to w hat others say
Think about how others see things
Ask others questions to get ideas
Stay out of fights
Keep anger under control
Stay out of trouble
Ask for help if needed
Program helped make/keep friends
Program helped make better decisions
Since coming to prog., relations w / family adults better
Talk about thoughts and feelings
Would talk to program staff about problem
Program staff expect me to try my best
Program staff care about w hat happens to me
Program helped identify caring adults
Program staff listen to me
Program staff act like young people are important
Young people can make difference at program
Think about how decision w ill affect my future
Think about how decision w ill affect others
Listen to others, then make ow n decision
Think about how made decision afterw ards
Think about consequences of decision afterw ards
C aring A dult R elatio nships
Staff
R elatio nships
P ro gram
Effectiveness
So cial Skills
D ecisio n M aking
Self C o ntro l
Em pathy
C o m m unicatio n
1
2
3
4
5
6
0.094
0.156
0.038
0.070
0.073
0.120
0.064
0.143
0.214
0.135
0.200
0.131
0.567
0.724
0.742
0.616
0.681
0.761
0.724
0.657
0.104
0.000
0.164
-0.017
0.008
0.034
0.044
0.647
0.540
0.559
0.225
0.280
0.131
0.197
0.372
0.137
0.033
-0.112
0.120
0.121
-0.026
0.100
-0.323
0.111
0.176
0.074
0.167
0.830
0.647
0.715
0.121
0.735
0.659
-0.174
0.035
0.277
0.576
0.163
0.041
0.195
0.031
-0.043
0.090
0.305
0.182
0.026
0.149
-0.034
0.045
0.067
0.006
0.094
-0.020
0.189
0.040
0.094
0.678
0.583
0.279
0.722
0.467
0.099
0.054
0.219
0.113
0.194
0.107
-0.077
0.045
-0.008
0.041
-0.025
0.169
0.073
-0.060
0.069
0.102
-0.014
0.128
0.240
0.095
-0.019
0.065
0.354
0.095
0.359
0.015
0.012
0.169
-0.003
-0.128
0.318
0.048
0.012
0.023
0.109
0.084
0.171
0.152
0.320
0.266
0.529
-0.014
0.077
0.137
-0.016
0.145
0.668
0.134
0.398
0.173
0.502
0.175
-0.049
0.108
0.210
0.074
0.009
0.129
-0.121
0.128
0.104
0.241
0.402
Factor Analysis - 13-19 year olds
ROTATED COMPONENT MATRIX: AGES 13+ (n=248)
O UT C O M E
C O N S T R UC T
IT E M S
Usually feel connected
Listen to w hat others say
Think about how others see things
Ask others questions to get ideas
Stay out of fights
Keep anger under control
Stay out of trouble
Ask for help if needed
Program helped make/keep friends
Program helped make better decisions
Since coming to prog., relations w / family adults better
Talk about thoughts and feelings
Would talk to program staff about problem
Program staff expect me to try my best
Program staff care about w hat happens to me
Program helped identify caring adults
Program staff listen to me
Program staff act like young people are important
Young people can make difference at program
Think about how decision w ill affect my future
Think about how decision w ill affect others
Listen to others, then make ow n decision
Think about how made decision afterw ards
Think about consequences of decision afterw ards
C a ring A dult R e la t io ns hips
Staff
P ro gra m
R e la t io ns hips
E f f e c t iv e ne s s
S o c ia l S k ills
S e lf C o nt ro l &
C o m m unic a t io n
E m pa t hy
D e c is io n M a k ing
1
2
3
4
5
0.076
0.134
0.026
0.017
0.086
0.185
-0.016
0.135
0.240
0.122
0.149
0.249
0.473
0.713
0.744
0.681
0.689
0.763
0.716
0.596
0.352
0.527
0.292
0.085
0.730
0.659
0.566
0.045
0.008
-0.003
-0.121
0.007
-0.057
0.251
0.106
-0.057
0.101
0.180
-0.037
0.399
0.300
0.171
-0.004
-0.025
-0.069
0.496
0.631
0.042
-0.139
0.155
0.184
0.158
0.117
0.371
0.226
-0.142
0.128
0.072
0.109
0.151
0.001
0.183
0.020
0.197
0.045
0.118
0.588
0.613
0.518
0.750
0.572
0.180
0.035
0.192
0.113
0.114
0.101
-0.035
0.203
-0.051
-0.032
0.080
0.030
0.634
0.626
0.650
0.291
0.364
0.132
0.158
0.386
0.185
0.043
-0.078
0.053
0.160
0.000
0.137
-0.226
0.611
0.064
0.142
0.119
0.678
0.268
0.239
-0.028
0.531
0.115
0.014
0.060
0.080
0.143
-0.011
0.222
0.000
0.256
0.181
0.088
0.399
Factor Analysis - Reliability
ALPHA COEFFICIENTS FOR FACTOR COMPONENTS: ALL CASES (n=389)
FACTOR
Staff
P rogram Effectiveness
Relationships
a
0.856
0.609
Self Control
Empathy
Communication
Decision M aking
0.659
0.584
0.578
0.640
ALPHA COEFFICIENTS FOR FACTOR COMPONENTS: AGES 13+ (n=248)
F A C T OR
a
Staff
R elatio nships
P ro gram Effectiveness
Self C o ntro l &
Empathy
C o mmunicatio n
D ecisio n M aking
0.845
0.614
0.648
0.660
0.707
Phase II: Con’t

Slight revision of individual items and rearrangement of
questions leading to final instrument:
–
–
–

Pencil and paper survey
40 questions addressing four core outcomes, program
participation, connectedness to program, and
sociodemographic information
Requires between 10 and 15 minutes to complete
Development of training modules, scoring templates,
and score report generating software
Phase III: Dissemination in
Rochester - May 2002 - May 2003



Summer and Fall 2002 - 11 youth serving
agencies in the Rochester area were trained to
use the instrument and the report generating
software
TA provided to agencies to develop appropriate
sampling plans
During the program year of 2002-2003, over
1,000 youth participating in YD programs in the
Rochester area completed surveys
Current Steps: May 2003 - present




Software and score reports revised based on
qualitative feedback from Year 1
implementation
Continued training and TA to current users
Dissemination to various other youth
development programs through ACT for Youth
Center of Excellence
Validation studies
Additional sites







Alaska - Residential School System served as a beta test site in
school year ‘02-’03
Erie County, NY - over 75 programs including youth bureaus and
youth boards, and UW and Department of Youth Services funded
programs
Hawaii - Children’s Alliance of Hawaii
Oswego County, NY - Oswego City-County Youth Bureau funded
programs
Salamanca, NY - 21st CCLC program
Syracuse, NY - Catholic Charities of Onondaga County sites
Queens, NY - Queens Child Guidance Center Beacon & OST sites
Use of Data by Programs

Internal quality improvement. Examples include:
–
–
–
–
–



Reviewing and discussing score reports with staff and with Boards
Comparing program scores within one agency to identify opportunities
for improvement
Reviewing curricula and current programming strategies
Discussing program strategies with other similar programs
Identifying training and technical assistance needs
Reporting to funders
Proposal writing
Sharing data back allowing the creation of an
aggregate community level score report using deidentified data
Training and Technical Assistance



Training, TA, and use of the READY Toolkit are
available to interested users
Fees are based on the number of users and the
level of training and TA required
Options include:
–
–
Training and TA provided directly to end users
One time training provided for end users, and
continuous training and TA provided to a lead agency
which then agrees to provide first line TA to end users
READY Toolkit

READY Toolkit includes a CD which contains:
–
–
–
–



A Personalizable Instrument Template
READY Analysis Program
Toolkit Instructions Manual
User’s Agreement
Instrument template may be personalized to contain
program names and staff titles
READY Analysis program allows community programs
to enter their own survey data, and generate a score
report
Score report contains summary measures for core YD
outcomes and frequencies for all survey items
SAMPLE
Score
Report
Pg 1 of 17
SAMPLE
Score
Report
Pg 2 of 17
Publications


Klein JD, Sabaratnam P, Matos Auerbach M, Smith SM, Kodjo C,
Lewis K, Ryan S, Dandino C. Development and factor structure of a
brief instrument to assess the impact of community programs on
positive youth development: The Rochester Evaluation of Asset
Development for Youth (READY) tool. Journal of Adolescent Health 2006;
39: 252-260.
Sabaratnam P, Klein JD. Measuring youth development outcomes for
community program evaluation and quality improvement: Findings from
dissemination of the Rochester Evaluation of Asset Development for
Youth (READY) Tool. Journal of Public Health Management and
Practice 2006; 6(suppl): S88-S94.
For more information about the
READY tool, please contact:
Premini Sabaratnam, MPH
Sr. Health Project Coordinator
Div. of Adolescent Medicine, University of Rochester
(585) 273-4616
[email protected]
or
Jonathan D. Klein, MD, MPH
Associate Professor of Pediatrics and of Community & Preventive Medicine
University of Rochester
(585) 275-7760
[email protected]
Or visit...
The University of Rochester, Division of Adolescent
Medicine, Leadership Education in Adolescent Health
website at
www.urmc.rochester.edu/gchas/div/adol/leah/resources.htm
or
The ACT for Youth, Center of Excellence website at
www.actforyouth.net