Transcript Slide 1

Kentucky 21st Century
Community Learning Centers
Fall 2013:
Multi-State Conference
Program Director’s Meeting
Site Visit, PQA Results and Next
Steps
Center for Evaluation & Education Policy
Indiana University
© David P. Weikart Center for Youth Program Quality
1
Presentation Overview
1. Introduction
2. Spring 2013 Site Visit Results
3. 2013 PQA Results
4. Quality Improvement System and Next
Steps
© David P. Weikart Center for Youth Program Quality
2
Spring 2013 Site Visits
Who? What?
•
28 visits to Cycle 9 KY 21st CCLC programs
February 18 through April 18, 2013
How?
1. Site coordinator interview
2. School day teacher interview
3. Standardized observation protocol for academic and
enrichment activities
© David P. Weikart Center for Youth Program Quality
3
Site Visit Rating System
Rating System:
•
12 Items
(rated on a scale of 1 to 4)
1 = Must Address and
Improve
2 = Some Progress Made
3 = Satisfactory
4 = Excellent
•
48 Possible Points
© David P. Weikart Center for Youth Program Quality
4
Site Visits (CEEP)
Purpose of 2013 Site Visits
Elementary/Middle
School Programs
High
School Programs
1. Activities geared toward
rigorous academic enrichment
1. Activities promote academic
growth, remediation, and
development
2. Links to the regular school day
2. Links to the regular school day
3. Individual support and
opportunities for positive
interactions for youth
3. Participants contribute ideas,
make choices, and having
positive experiences
4. Relationships with schools,
parents, and other community
constituents
4. Establish partnerships and
employ successful recruitment
strategies
© David P. Weikart Center for Youth Program Quality
5
Average Ratings of the Four Focus Areas
N=21
Focus Area (12 points possible in each area)
Elementary/Middle
School Sites
1 ) Activities Geared Toward Rigorous
Academic Achievement
9.3
2) Program is Linked to Regular School Day
10.1
3) Program Provides Individual Support and
Positive Interactions
9.6
4) Program Builds Relationships with
Community Stakeholders
10.6
© David P. Weikart Center for Youth Program Quality
6
Average Rating of Each Item within the Four
Focus Areas
Item
Elementary/Middle Sites
2013 Average Rating
(Out of 4 Possible
Points)
Focus Area 1: Rigorous Academic Achievement
High Quality Homework Help
Supplemental Academic Enrichment
Opportunities for Active Learning
3.1
3.1
3.1
Focus Area 2: Links to the School Day
Links to School Day Curriculum/School Standards
School Personnel Actively Involved with Program
Academic Activities Tailored to Individual Student Needs
3.4
3.5
3.2
Focus Area 3: Individual Support and Positive Interactions between Staff and Youth
Opportunities for Individual Support
Opportunities for Positive Interactions with Adults
Opportunities for Positive Interactions with Peers
3.2
3.3
3.1
Focus Area 4: Partnerships with Schools, Parents, and Community Constituents
Well-Integrated with School and Shares School Resources
Program Staff Initiate Regular Communication with Parents
Program Staff Develop Relationships with Community Partners
© David P. Weikart Center for Youth Program Quality
3.7
3.4
3.5
7
How to Target Rigorous Academic
Achievement in Elementary/Middle Schools
High Quality HW- Help – Staff
Supplemental Academic
circulate and work one-on-one with Enrichment – Include activities
students, homework alternatives
are available, maintain an
environment conducive to
homework completion (i.e. minimal
noise/adequate space)
with clear academic content, links to
academic content are explicitly
explained by staff, ongoing units
centered around themes or skill sets
Opportunities for Active
Learning – Provide activities that
include youth creating and reforming
tangible products, applicable to the
“real world”, links to STEM concepts
and incorporate reflection
© David P. Weikart Center for Youth Program Quality
8
Average Ratings of the Four Focus Areas
(N=7)
Focus Area (12 points possible in each area)
High
School
Sites
1 ) Activities promote academic growth, remediation, and
development
9.9
2) Program is Linked to Regular School Day
10.0
3) Participants contribute ideas, make choices, and have
positive experiences
10.2
4) Program establishes partnerships and employs
successful recruitment strategies
10.4
© David P. Weikart Center for Youth Program Quality
9
Average Rating of Each Item within the Four
Focus Areas
Item
High School Sites
2013 Average Rating
(Out of 4 Possible
Points)
Focus Area 1: Activities promote academic growth, remediation, and development
High Quality Homework Help/Tutoring
Credit Recovery/Remediation
Goal Setting/Career Development/Life Skills
3.4
3.5
3.0
Focus Area 2: Links to the School Day
Links to School Day Curriculum/School Standards
3.0
School Personnel Actively Involved with Program
3.7
Academic Activities Tailored to Individual Student Needs
3.3
Focus Area 3: Participants contribute ideas, make choices, and have positive
experiences
Opportunities for Interest-Based Choices
3.6
Youth are Engaged in Program Decisions and Development
3.3
Opportunities for Positive Interactions with Adults and Peers
3.3
Focus Area 4: Program establishes partnerships and employs successful recruitment
strategies
Intentional Student Recruitment and Retention Strategies are Used
3.0
Well Integrated with School and Shares School Resources
3.7
Parents and Community Based Organizations are Actively Engaged
3.7
© David P. Weikart Center for Youth Program Quality
10
High School Site Visit Results
Cross Year Comparison of Activities Offered by Sites
Activity
Percent of Sites Offering Activity
2012 (N=8)
2013 (N=7)
Homework
Help/Tutoring
100%
100%
Credit Recovery
43%
71%
Goal Setting/Career
Development
83%
71%
© David P. Weikart Center for Youth Program Quality
11
st
21
Kentucky
CCLC
2013-2014
Action Plan
© David P. Weikart Center for Youth Program Quality
12
Federal Recommendations for Kentucky 21st CCLC Programs
1. Kentucky must demonstrate that systems are in
place for quality control and technical assistance.
2. Programs must demonstrate
that data are being used for
continual improvement.
3. Programs must demonstrate
increased ability to achieve
results.
© David P. Weikart Center for Youth Program Quality
13
What is program quality?
Program
staff
students
???
outcomes
training
youth program
Another way to say it…
What do we want to see in high quality youth programs?
© David P. Weikart Center for Youth Program Quality
14
What is the Program Quality Assessment?
1. A validated instrument
designed to assess the quality
of youth programs and identify
staff training needs.
Engagement
Interaction
Supportive Environment
2. A set of items that
Safe Environment
measures youth access
to key developmental experiences.
3. A tool which produces scores that can be used for
comparison and assessment of progress over time.
© David P. Weikart Center for Youth Program Quality
15
PQA: The Pyramid of Program Quality
Plan
Make choices
Engagement
Reflect
Lead and mentor
Be in small groups
Partner with adults
Experience belonging
Interaction
Encouragement Reframing conflict
Supportive
Skill building
Session flow
Active engagement Welcoming atmosphere
Environment
Psychological and emotional safety
Program space and furniture Emergency procedures
Safe
Healthy food and drinks
Physically safe environment
Environment
Youth Voice and Governance
Professional Learning Community
© David P. Weikart Center for Youth Program Quality
16
Quality Improvement System (QIS)
Self Assessment Data
 A self assessment team (usually a program director and
site coordinator) observed two program activities.
 Activities observed included an academic enrichment and
social/cultural enrichment activity.
 Each site entered data from self assessment forms into
the online system Scores Reporter.
 Sites used a self-assessment form for school age youth or
older youth, depending on the grade levels of students
served.
© David P. Weikart Center for Youth Program Quality
17
Quality Improvement System (QIS)
Self Assessment Observation Scores
Keep in mind…
 Observation scores represent a snapshot – this has
limitations and value.
 These are aggregate scores from multiple observations.
 The overall story is more important than the individual
numbers.
 What you do with the data matters most!
© David P. Weikart Center for Youth Program Quality
18
Average Self Assessment Scores Assigned
Per Domain Across Sites
School Age (N=82)
Youth (N=76)
6
5
4.7 4.8
4.5
4.5
4
4.1
4.4
3.7
4
3
2
1
0
I. Safe
Environment
© David P. Weikart Center for Youth Program Quality
II. Supportive
Environment
III. Interaction IV. Engagement
19
Quality Improvement System (QIS)
External Assessment Data
 External assessments were conducted for 14 sites in
cycle 9 during the spring 2013 site visits.
 Sites in cycle 9 were randomly selected based on external
assessor availability.
 Assessments were conducted to give sites the opportunity
to compare self-assessment results to those conducted by
someone external to the program.
© David P. Weikart Center for Youth Program Quality
20
Quality Improvement System (QIS)
External Assessor Observation Scores
Keep in mind…
 External assessment scores are always lower than
self assessment scores.
 Observation scores represent a snapshot – this has
limitations and value.
 The overall story is more important than the individual
numbers.
 What you do with the data matters most!
© David P. Weikart Center for Youth Program Quality
21
Quality Improvement System (QIS)
How to target PQA Areas for Improvement:
Interaction— Structured opportunities for getting to know one another,
various types of groupings, opportunities to acknowledge the
achievements of the youth, youth have opportunities to mentor others, or
lead groups
Engagement— The youth have
opportunity to make plans, youth can
reflect on their activities, youth make
presentations to the whole group, youth
can make open-ended choices within
activities
© David P. Weikart Center for Youth Program Quality
22
2014 PQA Kentucky 21st CCLC
Self-Assessment Process: Tentative Timeline
MONTH
ACTIVITIES
•
PQA Basics training – introduce all Cycles to the tool and timeline for the PQA. All
new program directors and site coordinators who have not attended this training
in the past are required to attend. Marriot East, Louisville.

Self Assessment Team (Program Director and Site Coordinator), observes one
academic enrichment and one personal enrichment offering led by internal staff
and completes one PQA assessment for each offering observed.
February 6, 2014
February 12th-March
28th, 2014
Early/Mid March 2014 
March 28, 2014
May 2014
November 2014;
May 2015
Self Assessment Team attends a check-in/Scores Reporter webinar

All scores must be entered into online system Scores Reporter

All self-assessment teams (except for those teams who participated in previous
years) are required to attend the Planning with Data workshop held in May.
Location, Louisville. Date: TBD.
Self-assessment teams will complete a Program Improvement Plan and submit for
review.


Self-assessment teams will complete interim and final reports to document
progress made towards goals and objectives included in their Program
Improvement Plans.
© David P. Weikart Center for Youth Program Quality
23
Quality Improvement System (QIS)
What Happens Next?
 Year 2013 Participants:
 New program staff selected to be on the 2014 Self
Assessment team should attend the basics training.
 Those who have not yet participated in the QIS (cycle
10 expansion and new grantees)
 Required to enroll a Self-Assessment Team in the 2014
Quality Improvement System. Team members typically
include the Program Director and Site Coordinator.
© David P. Weikart Center for Youth Program Quality
24
Kentucky 21st Century
Community Learning Centers
Fall 2013:
Multi-State Conference
Program Director’s Meeting
Questions?
Stephanie Schmalensee, Research Associate
[email protected]
LeeAnn Sell, Evaluation Coordinator
[email protected]
© David P. Weikart Center for Youth Program Quality
25