Transcript Eight Steps To Quality Improvement

Karla A. Henderson, Ph.D. Professor
North Carolina State University
Deb Bialeschki, Ph.D. Senior Researcher
American Camp Association
Overview
National Youth • ACA National Outcomes Study
Development • ACA National Program
Improvement Project (with national
Research
Benchmark component)
Projects
Educational
Materials
• “Workbooks”
• Web-based tools and courses
American Camp Association (ACA)
ACA addresses human development for
children, youth, and adults
ACA is a community of camp professionals
dedicated to assuring quality in organized
camping
At least 10 million individuals are associated
with camp programs each year
ACA accredits over 2300 camps representing
different sponsorships, activity foci, and diverse
campers from across the United States
National Outcomes Study
Camper Growth Instruments (CGI)
• Pre-, post-, 6 month surveys with >5200 campers and their parents
• Pre- and post- counselors’ observation checklists
• Directors’ survey
Major Findings (Directions)
• Significant growth in all 4 domains (Positive Identity, Social Skills,
Physical & Thinking Skills, and Positive Values & Spirituality)
• Much of the growth was maintained over 6 months
• Parents indicated significant growth for their children in all 4
domains as a result of the camp experience as did the children’s
counselors
Needs That Emerged From The
National Outcomes Study
Need additional outcomes tools
•
•
•
•
Outcomes measures that are “modular”
Age-appropriate choices
Easy to administer and analyze surveys
Reliable and valid instruments
Need a “workbook” suitable for seasonal staff training
• Help staff connect their job responsibilities to outcomes
• “Recipe for success” (logic model)
• Facilitator guides
Need to look at how to provide an environment that supports positive youth development
ACA Camp Outcomes Surveys*
 Friendship Skills
 Teamwork
 Responsibility
 Competence
 Independence
 Exploration
 Family
Citizenship
*Reliable and valid
measures
Camper Learning




Camper Learning questionnaire (Interview Questions)
14 questions
Measures camp learning based on 7 domains
Examples:
 1.At camp, did you learn how to be better at making friends?
 2.At camp, did you learn to feel good about things that you do
well?
 3.At camp, did you learn that you can be good at some things that
you didn’t know you were good at?
 4 point scale:
I didn’t learn anything
about this
I’m not sure
I learned a little about
this
I learned a lot about
this
Older Camper: Basic Version
 Basic “Increase Format”
 Available for 7 outcomes
 6-14 items per outcome
 5 point scales
How much, if any, has your experience as a camper in this camp changed you in each of the
following ways?
Decrease
d
1. Placing group goals above the things
that I
want.
2. Working well with others.
Did not
increase
or
decrease
Increased
a little bit,
maybe
Increased
some,
I am
sure
Increase
d a lot, I
am sure
O
O
O
O
O
O
O
O
O
O
Older Camper: Detailed Version
 “Status + change format”
 Available for 7 outcomes
 6-13 items/outcome (status and change)
 6 point scales
1. I accept responsibility for my
actions
False
Somewhat
False
A little
False
A little
True
Somewhat
True
True
Is the above statement more or
less true today than before
camp?
A lot less
Somewhat
less
A little less
A little
more
Somewhat
more
A lot
more
False
Somewhat
False
A little
False
A little
True
Somewhat
True
True
A lot less
Somewhat
less
A little less
A little
more
Somewhat
more
A lot
more
2. I own up to my mistakes
Is the above statement more or
less true today than before
camp?
How do I know which one(s) to use??
 Young Camper (Camper Learning)
 Best for campers 7-9, possibly up to 10 or 11
 Shortest & fastest
 Unidimensional (camp learning)
 Basic Version




Campers are 10+
Interested in gains through camp
Less burdensome than status + change
Multidimensional (7 outcomes)
 Detailed Version





Campers are 10+
Interested in gains through camp
Interested in status after camp (correlations with other indices)
Most burdensome
Multidimensional (7 outcomes)
How Sherwood Forest Camps used
the Outcomes Measures
 During staff orientation
 Work by camper age groups
 Choose two outcomes for each village
 Measure those outcomes for at least two
summers
 With the campers
 In village meetings
 In cabin groups
 At teachable moments
Additional ACA Evaluation Resource: Creating
Positive Youth Outcomes Tool Kit
 A 7-step process for program evaluation to
help you increase positive youth outcomes. A
book and resource CD include:
 the evaluation outline,
 examples of goals and objectives you might
target,
 a facilitator's guide for staff training,
 copy-ready checklists and worksheets,
 forms for refining targeted outcomes, and
 youth survey tools designed and used by
ACA for youth outcomes research.
Benchmark & PIP Studies
Phase 1: Benchmarked supports and
opportunities perceptions of 7645 campers at
80 ACA accredited camps with YDSI survey
Phase 2: Program improvement year long
process with 23 of the 80 camps
Community Action Framework
for Youth Development
© Connell & Gambone 1998
Build Community
Capacity and Conditions
for Change (E)
Increase Supports
and Opportunities
for Youth (C)
Improve Youth
Development
Outcomes (B)
--Adequate nutrition,
health and shelter
--Multiple supportive
relationships
Implement Community
Strategies to Enhance
Supports and Opportunities
for Youth (D)
--Meaningful
involvement
--Challenging and
engaging skill building
activities
--Safety
Improve LongTerm Outcomes
in Adulthood (A)
A Word About Supports & Opportunities
Supportive
Relationships
Safety
Youth
Involvement
Skill-Building
• Guidance, emotional & practical support
• Adult & peers know who they are & what’s
important to them
• Physical & emotional
• Leadership
• Decision-making
• Belonging
• Challenging and interesting experiences that
build a wide array of skills
• Experience sense of growth and progress
Overall Supports and Opportunities
(Optimal) (Insufficient)
100%
80%
69%
60%
39%
40%
20%
30%
41%
25%
9%
1%
5%
0%
Supportive
Relationships
Safety
Youth
Involvement
Skill Building
Camp Characteristics
Dimension
Overall
Agency
Religious
Supportive
Relationships
62%
71%
78%
67%
47%
72%
Safety
28
35
34
25
22
31
Youth
Involvement
4
5
6
6
3
6
Skill-Building
35
41
51
36
27
42
2584
1338
1974
1756
883
6771
Sample
Size
IFP
INFP
Day
Resident
Camp Characteristics
Dimension
Overall
1 week
2-3 weeks
Supportive
Relationships
61%
66%
77%
66%
76%
72%
Safety
27
28
34
27
30
37
Youth
Involvement
4
5
7
5
6
6
Skill-Building
35
39
47
37
54
43
2410
2302
2923
4773
814
2101
Sample
Size
4+ weeks
Co-ed
Boys
Girls
Camper Characteristics
Dimension
Overall
10-11
yrs old
12-13 yrs
old
14-15 yrs
old
16-18 yrs
old
Supportive
Relationships
62%
69%
75%
82%
64%
72%
Safety
25
29
34
44
22
35
Youth
Involvement
3
4
7
15
4
6
Skill-Building
42
40
39
46
39
42
2366
3079
1749
451
2968
4586
Sample
Size
Male
Female
Camper Characteristics
Dimension
Overall
Supportive
Relationships
White
Non-white
1 summer
2-3
summers
4+
summers
72%
56%
59%
70%
76%
Safety
33
20
20
30
38
Youth
Involvement
6
3
3
5
7
Skill-Building
43
33
37
42
43
5946
1607
2263
2647
2644
Sample
Size
Phase 2: Program Improvement
 23 camps from original 80 benchmark camps
 6 month process:
 identify and set targets
 gather input from campers and staff
 conduct organizational assessment
 develop action plan
 set first year priorities
 Re-surveyed second summer
Eight Steps to PPI
Ask: Campers
Ask: Staff
Assess: Organizational Practices
Brainstorm: Strategies
Choose: Strategies and Develop Action Plan
Do: Take Action
Share: Action Plan
Evaluate: Take Stock
Organizational
Process
Improvement
Organizational structures (S):
• Low youth to staff/volunteer ratios
• Safe, reliable, and accessible activities and spaces
• Continuity and consistency of care
Organizational policies (P):
• Ongoing, results-based staff and organizational improvement
process
• Flexibility in allocating available resources
• Community engagement
Organizational activities (A):
• Range of diverse, interesting, and skill-building activities
• Youth engagement in organizational decision making
• High, clear, and fair standards
Findings About Strategies
 Grouped individual strategies into 7 categories:
 Mission- related
 Camp activities
 Programming/scheduling changes
 Staffing patterns
 Camper input & involvement in planning and
decision-making
 Staff training
 Facility
Major Findings- Phase 2 (PIP)
(Innovations, ACA, 2006)
 Significant improvement occurred in all areas
 Youth Involvement and Skill Building (83% in one or
both areas)
 More than 33% also strengthened in Supportive
Relationships and Safety
 Intentional, camper-centered assessment and
planning was important to positive change
 Camps with changes across all three organizational
practices (structures, policies, activities) were twice as
successful at producing change than if only 1-2 areas
were addressed.
Percent ACA PIP Camps with Positive
Change in 1 or More Dimensions
80%
70%
60%
50%
40%
30%
20%
10%
0%
Supportive
Relationships
Safety
Skill Building
Youth Involvement
Rate of Improvements Associated
with Level of Implementation
(Structure, Policy and Activities)
70%
60%
50%
40%
30%
11 S, P & A Camps
12 non SPA Camps
20%
10%
0%
Change in 3 or more
S&O areas
Change in 1 or 2 S&O
areas
No Change
Learned When Planning For Change…
1. Start with organization specific data
2. Use a youth development framework
3. Focus on intentionality and
accountability
Resources to help…
 Research-based (ACA’s youth
development “supports and
opportunities” study and
Program Improvement Project)
 Focuses on a step-by-step
process for organizational
improvement (across structures,
policies, and activities)
 On-line web-based support
course suitable for staff training
Future Research Ideas…
 Staff training around quality improvement (age and seasonality







of staff)
Parent understanding/perceptions about camp programs and
their children’s growth
Issues of intensity and dosage of the camp experience
Effect of the nature-based setting
Issues related to age/stage & optimal developmental supports
Relationships of camp experience on work force skills
development
“Culture of camp” within changing societal demographics
Role of camp experience within mental/physical health
perspective (child stress, physical activity, healthy lifestyles, etc)
Questions?
 Contact Deb Bialeschki at [email protected]
 Go to www.acacamps.org/research